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MEASURING VALUE AND EFFICIENCY: HOW TO ASSESS 
THE PERFORMANCE OF THE DEFENSE ACQUISITION 
SYSTEM 


House of Representatives, 

Committee on Armed Services, 

Panel on Defense Acquisition Reform, 
Washington, DC, Wednesday, April 1, 2009. 

The panel met, pursuant to call, at 7:33 a.m., in room 2212, Ray- 
burn House Office Building, Hon. Robert Andrews (chairman of the 
panel) presiding. 

OPENING STATEMENT OF HON. ROBERT ANDREWS, A REP- 
RESENTATIVE FROM NEW JERSEY, CHAIRMAN, PANEL ON 

DEFENSE ACQUISITION REFORM 

Mr. Andrews. Ladies and gentlemen, good morning. The panel 
will come to order. 

I am informed that we expected our ranking member, Mr. Con- 
away, will be present shortly. But because one of our minority col- 
leagues is present, we are going to begin. 

First of all, I appreciate the indulgence of the witnesses and my 
colleagues and our staff in being here at such an early hour. I hope 
that we did not inconvenience people terribly too much. 

The reasoning behind this is that this is very substantive and 
important material. And we want the members to be able to have 
an uninterrupted block of time to really hear what the witnesses 
say, to engage in what I hope would be constructive dialogue with 
the witnesses, and not be caught up in our normal time pattern 
around here, which is the bell ringing to vote and conflicting with 
other hearings and meetings. 

We really want to give our utmost and most serious attention to 
the material, so that is the reason for this early beginning. And I 
appreciate the indulgence of the members of the panel. 

This morning we are setting out to try to answer the question, 
what is the fair measure, what is the fair way of measuring the 
difference between — the difference, if any — between the cost paid 
by the taxpayers to acquire goods and services in the Department 
of Defense (DOD) and the value we are receiving? Is there a dif- 
ference between those two concepts? I think there is. And if there 
is, what is a fair way of measuring that difference? 

We have two outstanding witnesses this morning who can speak 
with great authority to that question. One, Mr. Sullivan, in his 
work at the Government Accountability Office (GAO), has frankly 
already given us a compelling measure of the answer to that ques- 
tion with respect to major weapons systems. He is going to talk 
this morning about the most recent work the GAO has done. 

( 1 ) 
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And here is essentially what it says. In 2003, we had 77 major 
weapons systems that were subject to this kind of evaluation. We 
now have 96. 

In 1977, the average cost differential — the increase in programs 
over their original base line — that is a very important concept, the 
original base line versus the adjusted one. But their cost increase 
over their original base line in 2003 was 19 percent in the aggre- 
gate. That number has gone the wrong way since 2003. It has now 
gone up to 25 percent in the most recent data that the GAO has 
presented. 

What is interesting about that 25 percent, it happens to be one 
of the triggers in the Nunn-McCurdy legislation that was passed 
quite a few years ago now, it is one of the triggers that triggers 
a very intense level of scrutiny of a major weapons system. So it 
is kind of discouraging to think that, looked at in the aggregate 
across these 96 weapons systems, in the aggregate, they now all 
trigger this kind of more intense assessment. 

Now, these data, as all good work does, really beg a different set 
of questions. Why? 

And in looking at the testimony this morning, and from hearing 
witnesses, I am sure we are going to hear that there is a lot more 
to this than meets the eye. 

The superficial response to this would be to say, “Well, geez. The 
people who are building these systems and managing them must 
be doing a really terrible job.” That is not necessarily so. 

And I think what we are going to hear this morning is, if you 
go deeper into the process, you find two other questions that have 
to be looked at. 

The first is how good or bad of a job we are doing at conceiving 
these systems in the first place. When there is a need identified, 
and there is a weapon system identified to fill that need, are we 
following the right process to determine what should fill the need? 

What in the jargon is called an AO A, which is an independent 
Analysis of Alternatives, how well or poorly are we doing that? Be- 
cause if we do a good job with the analysis of alternatives, we pre- 
sumably go down the right path to fill the need and provide the 
capabilities that the service members need. 

So, one of the questions I think lies below the disturbing data 
with which we start this morning is: How effective is that AOA sys- 
tem? 

And then the second goes to the question of how accurate the 
original base line is, or how inaccurate the original base line is. It 
is certainly not fair to blame those who are implementing a weap- 
ons systems, if the standards against which they are being meas- 
ured were unrealistic and flawed in the first place. That may or 
may not be the case, but it is something else, again, I think that 
we are going to hear about. 

There is significant evidence to show that the huge adjustments 
from the original base line to the modified ones may not be a meas- 
ure of a lack of aptitude by those implementing the systems. It 
may be a measurement of lack of accuracy by those establishing 
the original base lines. 
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So, the standard against which we are measured is a very impor- 
tant question. And frankly, it appears like we do not have the tools 
to answer that question particularly well. 

The final point that I want to make is that we also want to go 
beyond this morning’s discussion. This morning’s discussion, by ne- 
cessity, focuses on major weapons systems, major weapons acquisi- 
tion. 

But as we heard last week in the briefing, a significant percent- 
age — at least half — of the procurement done under the Department 
of Defense is not major hardware systems, it is services. And we 
want to be sure that we are in a position to take a comprehensive 
look at those issues as they come along, as well. 

I am glad that my friend and copilot has arrived. And if he has 
had a chance to catch his breath, I would be happy to yield to him 
and ask him for any introductory comments. 

Good morning, Mike. 

[The prepared statement of Mr. Andrews can be found in the Ap- 
pendix on page 33.] 

STATEMENT OF HON. K. MICHAEL CONAWAY, A REPRESENTA- 
TIVE FROM TEXAS, RANKING MEMBER, PANEL ON DEFENSE 

ACQUISITION REFORM 

Mr. Conaway. All right. Thank you, sir. My apologies. The doors 
down in front were not open, and the police had a long rollcall as 
their excuse. 

Mr. Andrews. So much for bipartisanship. Now we lock the 
doors on these guys. [Laughter.] 

But we locked a democrat out, too? Okay. All right. [Laughter.] 

Mr. Conaway. Yes, he went the other way. 

Good morning, Mr. Chairman, ladies and gentlemen. 

Mr. Andrews. Good morning, Mike. 

Mr. Conaway. I think it is appropriate for the panel’s first hear- 
ing that we have a senior member of the Defense Acquisition Sys- 
tem and GAO’s senior acquisition management professionals sitting 
side by side. It is not that often that members of this committee 
get a chance to talk to the Department of Defense and GAO at the 
same time. 

Thank you, gentlemen, for making this possible and agreeing at 
such an early hour. 

The first question this panel identified as part of the work plan 
was whether there is a method to reasonably measure the ability 
of the defense acquisition system to deliver the goods and services 
needed by the warfighter, and to do so in a timely fashion, and to 
do so at a fair price to the taxpayer. 

Today’s hearing will likely not answer the larger philosophical 
question about how one should measure value in defense acquisi- 
tion, but is an important first step for us to understand how DOD 
and GAO currently assess performance in one segment of defense 
acquisition, the major weapons systems programs, that were the 
focus of GAO’s assessment released this week. 

These programs receive a great deal of scrutiny by Congress and 
by the media for good reason. GAO’s report reveals that nearly 70 
percent of DOD’s 96 largest weapon programs were over budget 
$296 billion, or 42 percent. This is simply unacceptable. 
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Everyone understands why we cannot continue to tolerate these 
cost increases. There is little more to be said on that subject. 

But what we do not hear as much about is that the GAO had 
encouraging words to say about the steps the Pentagon has taken 
to improve acquisition outcomes, including early stage systems en- 
gineering, prototyping, measurable yearly plans, increasing ac- 
countability and minimizing requirements creep. 

The report states, these changes are consistent with a knowl- 
edge-based approach to weapons development that we have rec- 
ommended in our work. If implemented, these changes can help 
programs to reduce risk with knowledge, thereby increasing the 
chances of developing weapons systems within cost schedule tar- 
gets, while meeting user needs. 

These are encouraging signs. But to improve outcomes on the 
whole, DOD must ensure that these policy changes are consistently 
implemented and reflected in decisions on individual programs. 

I hope we hear more today about these positive improvements 
that DOD is making and what more needs to be done. Of course, 
we are likely to learn that much of what DOD does to measure per- 
formance is already statutorily required. 

I also hope our witnesses feel free to share their views on laws 
and regulations that are not assisting in their efforts to obtain the 
best value and capability for our warfighters. There is a balance to 
be struck between setting high expectations and over-regulating 
the system. 

With that I conclude, and again thank my fellow members. 

And Mr. Chairman, I look forward to the witnesses’ testimony. 

[The prepared statement of Mr. Conaway can be found in the Ap- 
pendix on page 34.] 

Mr. Andrews. Thank you, Michael, very much. 

And just echoing my friend’s opening statement, when he talks 
about the $296 billion in overruns, it is interesting from Mr. Sulli- 
van’s testimony we are going to hear that the weapons systems I 
made reference to have a total projected cost of $1.6 trillion. And 
half of that money is yet to be expended. 

So, to put that in some context, if the 25 percent overrun that 
the GAO now reports had not been the case, the 25 percent of that 
$800 billion is $200 billion. You know, the deficit this year is about 
$1.2 trillion. It is a sixth of the deficit that we are talking about, 
just from these systems expressed in one-year terms. So, it is a lot 
of money and is of great significance. 

Without objection, any opening statements from other panel 
members will be included in the record. 

I want to first go through the biographies of the witnesses. 

And we are going to ask the witnesses — without objection, we 
have entered your written testimony into the record of the panel — 
to summarize your testimony in about five minutes. We are not 
going to rigidly adhere to that rule this morning, but we would like 
you to try to summarize your testimony in about five minutes, and 
then we will proceed to questions from the members. 

David G. Ahern is the Director of Portfolio Systems Acquisition. 
He is responsible for providing portfolio management, technical and 
programmatic evaluation and functional oversight. His office sus- 
tains Department of Defense strategic and tactical programs in 
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support of the Under Secretary of Defense for Acquisition, Tech- 
nology and Logistics, and the Deputy Under Secretary of Defense 
for Acquisition and Technology. 

Mr. Ahern was previously professor of program management and 
Director of the Center for Program Management at the Defense Ac- 
quisition University (DAU) at Fort Belvoir, Virginia. While at 
DAU, Mr. Ahern also served as an executive course learning team 
mentor and instructor at the Defense System Management College, 
School of Program Management. 

Mr. Ahern has also held business development, program manage- 
ment and business unit positions in the development of tactical in- 
formation systems with General Dynamics Information Systems 
Company and the Northrop Grumman Electronic Systems sector. 

A native of Connecticut — one of the Final Four participants — Mr. 
Ahern was a career naval officer and is a graduate of the Naval 
Academy in Annapolis. He is also a graduate of the Naval Post- 
graduate School and Defense Systems Management College. Mr. 
Ahern’s sea duty was as a naval aviator in the RA-5C Vigilante 
during multiple deployments in the Pacific and Atlantic, and as an 
Executive and Commanding Officer of Tactical Electronic Warfare 
Squadron 33. 

Ashore, he was head. Tactical Command and Control Branch on 
the staff of the Chief of Naval Operations, project officer of the 
Navy Space Project, Class 2 Program Manager at the Joint Tactical 
Information Distribution System (JTIDS) Program Office, Program 
Manager, Navy Tactical Data Link Systems, and Deputy of the 
Program Executive, Office Space, Communications and Sensors. 

Mr. Ahern, thank you, and it is great to have you with us this 
morning. 

Mike Sullivan — no stranger to this committee — serves as Direc- 
tor, Acquisition and Sourcing Management at the U.S. Government 
Accountability Office. This group has responsibility for examining 
the effectiveness of agency acquisition and procurement practices 
and meeting their mission performance objectives and require- 
ments. 

In addition to directing reviews of major weapons system acquisi- 
tions, Mr. Sullivan has developed and directs a body of work exam- 
ining how the DOD can apply best commercial practices to the na- 
tion’s largest and most technically advanced weapons systems. 

This work has spanned a broad range of issues critical to the suc- 
cess in the delivery of systems, including quality assurance, transi- 
tion to production, technology inclusion, requirement setting, de- 
sign and manufacturing, reducing total ownership cost, software 
management and affordability. His team also provides the Con- 
gress with early warning on technical and management challenges 
facing these investments. 

Mr. Sullivan has been with the GAO for 23 years. He received 
a bachelor’s degree in political science from Indiana University and 
a master’s degree in public administration from the School of Pub- 
lic and Environmental Affairs in Indiana University. Mr. Sullivan 
is married and has two children. 

Welcome, gentlemen. We are really happy to have you with us. 

And Mr. Ahern, we will start with your testimony. 
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STATEMENT OF DAVID G. AHERN, DIRECTOR OF PORTFOLIO 

SYSTEMS ACQUISITION, OFFICE OF THE UNDER SECRETARY 

OF DEFENSE FOR ACQUISITION, TECHNOLOGY AND LOGIS- 
TICS 

Mr. Ahern. Thank you, and good morning, Chairman Andrews, 
Ranking Member Conaway, distinguished members of the panel. 

Thank you for the opportunity to appear before you today to dis- 
cuss how the department values the acquisition programs and as- 
sesses the effectiveness of ongoing developments in procurements. 
I will be brief in order to move quickly to the panel’s questions. 

In December of 2008, the department issued a new version of the 
DOD Instruction 5000.2, Operation of the Defense Acquisition Sys- 
tem. This instruction established policies and procedures for all of 
the department’s acquisition programs. 

It provides for a structured, disciplined process and incorporates 
many initiatives aimed at improving not only the defense acquisi- 
tion system as a whole, but also execution of individual programs. 
I would like to take a few minutes to highlight a few of the initia- 
tives. 

First, the department has established a mandatory — a manda- 
tory — materiel development decision (MDD) review that represents 
the formal entry point into the acquisition system. Every program 
will go through that MDD. At the Materiel Development Decision, 
the preliminary concept of operations, a description of the needed 
capability, the operational risk and the basis for determining that 
a non-materiel approach will not sufficiently mitigate the capability 
gap are thoroughly discussed. 

Also discussed is study guidance for the analysis of alternatives, 
which, when completed, will have examined the full spectrum of al- 
ternatives, starting with the current capability and moving to an 
entirely new materiel solution, with a goal of balancing capability 
needs with what the department can effectively acquire and afford 
to achieve the best value proposition for our nation. 

A second major change to the DOD 5000 is a revamp technology 
development phase. In the technology development phase the de- 
partment seeks to reduce technology risk, determine the mature 
technologies to be integrated into a full system and demonstrate 
critical technology elements on prototypes. 

Competitive prototypes, whether at the system or component 
level, reduce technical risk, validate designs and should improve 
cost estimates. They will also enable the evaluation of manufac- 
turing processes and, of course, refine requirements — again, with a 
goal of ensuring the acquisition enterprise pursues the best value 
solution to meet warfighter needs. 

Taken together, the Materiel Development Decision, a materiel 
solution analysis and a technology development phase with com- 
petitive prototyping help to define the best value acquisition pro- 
gram to meet the warfighter needs. 

We then execute the selected alternative in the Engineering and 
Manufacturing Development (EMD) phase, using additional tools to 
keep the program on track. 

The Acquisition Program Baseline is the key document for pro- 
gram management. It reflects the approved program being exe- 
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cuted. It describes the cost estimate, the schedule, performance, 
supportability and other relevant factors for the program. 

That Acquisition Program Baseline is the way that we track the 
progress of the program through the development in EMD and on 
into procurement. An acquisition strategy describes how the pro- 
gram manager plans to employ contract incentives to achieve re- 
quired cost, schedule and performance outcomes. 

Technical reviews are another tool the department uses to assess 
program status and for decision-making purposes. There are De- 
fense Acquisition Board reviews where the members advise the 
Under Secretary of Defense on critical acquisition decisions. 

Further, there are configuration steering boards held by the serv- 
ice acquisition executives that meet at least annually to review all 
requirements, changes and any significant technical and configura- 
tion changes in their programs that have the potential to result in 
cost and schedule impacts. Such changes will generally be rejected, 
deferring them to future blocks or increments. 

Program support reviews are means to inform the Milestone De- 
cision Authority and program office of the status of technical plan- 
ning and management processes by identifying cost, schedule and 
performance risk and recommendations to mitigate those risks. 

Defense acquisition executive summaries review programs 
monthly and provide an early warning report describing the actual 
program problems, warnings of potential problems and mitigation 
actions. 

In addition to the program level tools described above, the de- 
partment also employs mechanisms to monitor contract-specific 
performance. Earned value management is mandatory on cost and 
fixed-price incentive contracts above a certain low threshold value. 
It is a well-known tool used by both government and industry pro- 
gram managers to measure contract performance against a contract 
baseline, and it provides an early warning for baseline deviations 
and a means to forecast final cost and schedule on that contract. 

I have only touched on a few elements in the new DOD 5000, 
aimed at ensuring programs are started with a solid foundation, 
are focused on disciplined execution and deliver capability to the 
warfighter within cost and schedule parameters. 

It will take time for us to fully realize the benefit of these policy 
initiatives, and we will continue to look for opportunities to further 
improve the defense acquisition system. 

I look forward to the opportunity to work with the members of 
this panel on this critical task, and I am grateful to the members 
of this committee for your support for the Defense Department. 
Thank you. 

[The prepared statement of Mr. Ahern can be found in the Ap- 
pendix on page 38.] 

Mr. Andrews. Mr. Ahern, thank you for your service and your 
excellent work throughout your career and your contribution this 
morning. Thank you very much. 

Mr. Sullivan, welcome. Welcome back. 
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STATEMENT OF MICHAEL J. SULLIVAN, DIRECTOR FOR AC- 
QUISITION AND SOURCING MANAGEMENT, U.S. GOVERN- 
MENT ACCOUNTABILITY OFFICE 

Mr. Sullivan. Thank you, Mr. Chairman, Ranking Member Con- 
away, other members of the committee. I am pleased to be here 
this morning to discuss how best to measure the value DOD is pro- 
viding to the warfighter. 

Earlier this week, we reported that a cumulative cost growth in 
DOD’s portfolio of 96 major defense acquisition programs was $296 
billion, and the average delay in delivering promised capabilities to 
the warfighter was 22 months. 

These outcomes mean that other critical national priorities go 
unfunded, and warfighters go without the equipment they need to 
counter ever-changing threats. This should be unacceptable. 

A single metric or set of metrics is not enough to monitor acquisi- 
tions and gain efficiencies. However, a cross-cutting set of metrics 
that can measure knowledge, processes and outcomes can be em- 
ployed to improve acquisition outcomes. 

We think about metrics and their value in the following context. 

First, we use knowledge metrics to determine how well acquisi- 
tion programs are managing and retiring predictable technology, 
design and manufacturing risks by gaining knowledge and retiring 
risk. These metrics are valuable, because they can predict prob- 
lems, and they can identify the causes of those problems, so you 
can attack those causes. 

Second, we use cost, schedule and capability metrics that meas- 
ure a program’s health. These metrics have intrinsic value as sim- 
ple measurements, but they do little in the way of diagnosing cause 
and effect. This is a way for managers and decision-makers to keep 
an eye on the program. 

Third, there are certain indicators that we look for that are per- 
haps more important than the metrics, because they determine the 
realism of the acquisition plans from the outset, as the chairman 
was referring to in his opening statement. 

These are a set of prerequisites for any program. And without 
them, we question the value of any metric as you move forward. 

We know that the knowledge and program health metrics we use 
to measure a program’s progress and outcomes are valuable when 
used in realistic, schedule-driven product development environ- 
ments. They are important indicators to decision-makers. They 
work when they are measuring realistic plans and goals that are 
supported by doable requirements, appropriate cost and schedule 
estimates, and stable funding. 

Our knowledge metrics identify potential problems that could 
lead to cost and schedule shortfalls, and their likely causes. They 
identify technology readiness levels very early, measure design sta- 
bility by about midway through a development program and track 
whether critical manufacturing processes are in control at the start 
of production. 

They have predictive value. Generally, programs that do not 
measure these risks at the right junctures will encounter a cascade 
of problems beginning with design changes and continuing with 
parts shortages, changes to the manufacturing processes, labor in- 
efficiencies on the manufacturing floor and quality problems that 
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will cost money. All of these things delay programs and add to 
their costs. 

Outcome metrics provide useful indicators about the health of ac- 
quisition programs, and are valuable tools to improve oversight. 
Last year, the Office of Management and Budget (0MB) tasked 
DOD to work with us to develop a comprehensive set of outcome 
metrics, to track program costs and schedule performance and 
trends. We agreed to track trends and changes across eight dif- 
ferent cost and schedule data points, which are in my written 
statement — I will not go through them here — from each of the pro- 
gram’s, from their original baseline, from a five-year-out period and 
from a year ago. We do that on every program. 

These metrics give decision-makers, such as you, some visibility 
into the direction an acquisition may be heading in terms of cost 
and schedule. 

We scale these outcome metrics up from the individual programs 
to a portfolio level, to provide senior department leaders and the 
Congress with a snapshot of the cumulative impact of poor pro- 
gram performance on the relative health of the overall portfolio and 
which way it trends. 

For example, we know that the cost of the portfolio has doubled 
since 2000. There are 19 more major acquisitions in the portfolio. 

Development cost, as the chairman referred to, has grown by 42 
percent. And cost growth has forced the department to reduce 
quantities on many programs. Programs are getting less for their 
money, DOD’s overall buying power is reduced and less funding is 
available for other priorities. 

Metrics by themselves cannot be valuable unless the department 
does a better job ensuring that acquisitions start with realistic 
baseline estimates for cost and schedule. I think Mr. Ahern went 
through a lot of the initiatives in the new policies that we think 
are encouraging in this regard. 

We believe there is a set of prerequisites that must be a part of 
any acquisition strategy before any measurement of an acquisi- 
tion’s health can be valuable. Otherwise, metrics measured unreal- 
istic estimates will do no good. 

Quickly, these prerequisites include: number one, setting prior- 
ities by ensuring joint acquisitions more often and validating only 
candidates that are truly needed and feasible; number two, making 
a knowledge-based, evolutionary business case for the product; 
number three, separating technology development activities from 
product development activities, which we think is really key, be- 
cause if you get immature technologies onto these product develop- 
ment programs, they cause a lot of problems; number four, limiting 
the time and the requirements for product development to a man- 
ageable level; number five, employing systems engineering dis- 
cipline early to develop realistic cost and schedule estimates before 
product development starts; and committing to fully funding devel- 
opment programs once they are approved. 

Mr. Chairman, I will stop there and conclude my statement. I 
will be happy to answer any questions the committee may have. 

[The prepared statement of Mr. Sullivan can be found in the Ap- 
pendix on page 51.] 
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Mr. Andrews. Well, thank you, gentlemen, both, for very edi- 
fying and useful testimony. We appreciate the effort. And as I say, 
your full statements have been entered into the record. 

Mr. Sullivan, the title of your testimony is “Measuring the Value 
of DOD’s Weapon Programs Requires Starting with Realistic Base- 
lines.” And I think that that is a point that is made very forcefully 
in your testimony this morning and beyond that. And you highlight 
the importance of the realism of acquisition plans. 

I think it is important to note that, not only is that an important 
measurement tool, but it has everything to do with whether the 
Congress can make decisions based upon realistic assumptions. 
You know, if a system is sold to us on the basis that it is going 
to cost $1, and a realistic projection would be it is going to cost 
$1.25 or $1.40, it has a very different set of dynamics that would 
then take place in our decision-making. 

So, this is really a separation of powers issue, in a sense, that 
for us to make an intelligent, clear-eyed decision about what to do, 
we need better data on which to make that decision. 

What recommendations would you make? And I know you do this 
in your written testimony, but I would like you to elaborate. What 
recommendations would you make to improve the accuracy and 
transparency of the planning process, the standard-setting process 
that goes into these decisions? 

And secondly, to what extent do you think the new 5000.2 guid- 
ance moves us in that direction? 

Mr. Sullivan. Okay. First of all, I think the 5000 — the new 5000 
guidance — does a lot, I think, to move us in that direction. 

Where we would, as auditors, where we would — and people that 
are interested in oversight — we have problems with what is re- 
quired and what is suggested. And, you know, often times in those 
policies, there is a lot of wiggle room and a lot of encouragement 
to do things, but not a lot of requirements to do things, so we al- 
ways take issue with that. 

Having said that, the principles in that new policy — and Mr. 
Ahern went through some of that 

Mr. Andrews. Right. 

Mr. Sullivan [continuing]. Address a lot of the things that we 
think have to be in place to improve these weapon system 

Mr. Andrews. Tell us what some of them are. 

Mr. Sullivan. For example, at the beginning, the initial deci- 
sion — it escapes me for the moment 

Mr. Ahern. MDD. 

Mr. Sullivan [continuing]. The MDD decision is much more of 
a joint decision today. I think they have tried to bring in — you 
know, there are three big processes. There is a requirement-setting 
process, the funding process and then the acquisition execution 
process. And they have trouble speaking to each other a lot of 
times. 

I think there has been a real push to try to get them together, 
so that there is agreement, a joint agreement on how to move for- 
ward with a program. That is one way. 

The Analysis of Alternatives that you spoke of, I think the policy 
is trying to tweak the way that they analyze alternatives and try 
to — and is trying to bring a little more jointness into that, for ex- 
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ample, so that you do not have — in the past, an Analysis of Alter- 
natives was done by perhaps one of the services. 

And if it was the Army, they were looking at something that had 
wheels. And if it was the Navy, it was something that floated. And 
if it was the Air Force, it flew. And so, there were probably a lot 
of options that did not get full consideration in that. 

As you move to more jointness, I think you get a better idea 

Mr. Andrews. Do you think it is also characteristic in that AOA 
that there was not adequate consideration, simply revamping an 
existing system as opposed to starting all over again? 

Mr. Sullivan. Probably that was the case. And so now, the guid- 
ance does state that the number for — ^you know, when you have a 
new threat or something that you have to counter, the first thing 
you do is look at doctrine. You look at training, you look at a dif- 
ferent way, or modifying existing systems. 

You know, that is a tough one to answer. But I think a lot of pro- 
grams do get started that probably should not. You know, they 
probably could have found an alternative way to do things. 

Mr. Andrews. Are you satisfied that there is enough guidance on 
looking to the commercial world as a place to look for solutions to 
these needs that are identified, in the AOA process? 

Mr. Sullivan. I am not satisfied that they do enough of that. 

You know, usually when there is a — when they are thinking — 
well, I will give you an example of kind of commercial, off-the-shelf 
programs that the department has tried to start in the past. I 
think it is fair to say that programs like Warfighter Information 
Network-Tactical (WIN-T), which is a communications program, 
was designed to kind of take commercial, off-the-shelf items and 
modify them a bit, and make them available to the warfighter. And 
that did not work very well. 

There has been a lot of talk about the Presidential Helicopter 
and how that has gotten way out of control, because they were 
thinking that would be a commercial, off-the-shelf item. And it 
came in, often times 

Mr. Andrews. Who has been talking about that? I do not know. 

Mr. Conaway. First I have heard of it. 

Mr. Sullivan. The requirement-setting process has an impact on 
all that stuff. 

Mr. Andrews. Right. 

Mr. Sullivan. Once you have a feasible idea to do something 
commercially, and the users kind of start looking at it, all of a sud- 
den requirements start getting piled on that, and it becomes some- 
thing much different. 

Mr. Andrews. Well, thank you, gentlemen. 

I am going to turn to Mr. Conaway for his questions. 

Mr. Conaway. Well, thanks, gentlemen. I appreciate you both 
being here. 

You know, it is pretty presumptuous of us to think that we are 
going to ever know as much about acquisitions as the panel, or 
probably a bunch of you all sitting in the room. 

I guess our role, though, is to try to elicit from you the solutions, 
because I have got to believe that with the vast background that 
both of you have, and everybody else across the system has, nobody 
wants us in the position that we are in today. And so, having us 
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ask the right questions, I think is the best job that we can do in 
this circumstance. 

On the mandatory development decision, I guess, MDD, is there 
enough rigor there to make sure that the folks making the esti- 
mates are not simply — and this is a bit crass — ^but simply low-ball- 
ing the estimates in order to get the program started? Because 
once something is started and the initial inertia is overcome, it 
moves, whatever happens. 

And so, is there enough, you know, auditing or somebody check- 
ing the guys doing these assessments in that initial phase to say, 
you know, these estimates are not realistic? 

Mr. Ahern. I think you are talking to me, sir, congressman. 

The MDD process, we really do not get too hard into money. But 
as was described in my — into the funding that is going to be re- 
quired — that really is a transition period between the Joint Capa- 
bilities Integration and Development System (JCIDS) process, as 
Mr. Sullivan mentioned, and “little a.” 

Let me elaborate for a minute what I expect in the MDD. And 
I am absolutely a proponent of it. I think it is really critical. 

As Mr. Sullivan said, it is the place where we get both the re- 
source sponsors, the senior people from the Joint Chief of Staff 
(JCS), as well as ourselves, in a room and talk through what it is 
we are trying to do. And that is predicated on a good job over in 
the JCS arena of going through the Concept of Operations 
(CONOPS) for the system, whatever they are talking about, going 
through an Analysis of Alternatives themselves, a smaller Analysis 
of Alternatives. 

Is a materiel solution needed, or can we change training and doc- 
trine and get the job done, and that capability gap in that way? 

If a materiel solution is needed, then they neck it down. And 
they have an Initial Capability Document (ICD) that is actually ap- 
proved by the Joint Requirements Oversight Council (JROC), of the 
very senior group in the Joint Chiefs of Staff (JCS). 

So, coming into the MDD, the JCS has said, this is a capability 
gap that we need to fill, and it needs a materiel solution. 

In the MDD itself, with that as a starting point, and the JCS 
briefs that ICD what it is they want from a materiel solution, then 
we talk through what the Analysis of Alternatives needs to be. And 
that is where we kick off and start the Analysis of Alternatives. 

We have all had an opportunity to look at the guidance for the 
Analysis of Alternatives. And it is going to start. Chairman An- 
drews, with, can we modify the current system? That is absolutely 
already there. 

And then it will gradually go up in a ladder step, if you will, all 
the way to pressing technology to a really — got to choose my words 
carefully here, gentlemen — ^but you can have a number of alter- 
natives, one of which, the last of them could be we have got to push 
technology to get what it is. 

But then, in that Analysis of Alternatives, you have got to look 
at measures of effectiveness. How is it supposed to perform? What 
reliability are you expecting of it? What suitability are you expect- 
ing of it? What survivability are you expecting of it? 

And then you need to look at the costs. And so, what we will do 
in the MDD is, we will talk through that acquisition, the Analysis 
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of Alternatives plan. Is it rich enough? Is it robust enough? Is it 
considering the alternatives that you are talking about? 

And I have done one of these. I am not just making this up as 
I go along. We have been through one. 

I structured very carefully to be sure that we had that dialogue, 
because we have the senior stakeholders in the room, and we want 
to get the benefit of their advice and counsel as we are going for- 
ward. And then we will talk through it. 

Of course, a very significant part of the Analysis of Alternatives 

is, in fact, the cost analysis versus the various alternatives that we 
are talking about. If you are going to modify a system, it should 
be less expensive than pushing technology. And that needs to be 
compared. 

What are you going to get in terms of suitability and effective- 
ness versus cost? That is part of the outcome. And do you under- 
stand the environment that it is going to be operated in? So, we 
talk through the Analysis of Alternatives plan. 

And finally in that MDD, we give the program manager — and 
there is a designated program manager for all of them — an oppor- 
tunity to tell us what he thinks the whole program will look like. 
Now, that is early to need. 

But there will be funding associated, because, as you all know, 
we do a Program Objective Memorandum (POM) over a number of 
years. And there will be a rudimentary schedule — not prejudicing 
the AOA, because remember, the program has not started yet at 
the MDD. 

Following the MDD, we go to a milestone A, which is, as Mr. Sul- 
livan recognized, the beginning of the technology development 
phase. And following that technology development phase, then we 
will get into the product development phase. That is where we 
snap the chalk line and put down the basis of the program, of the 
cost estimate. 

But we do want to have the program manager talk to us in gen- 
eral terms about how they are looking at this program, how long 
it is going to take to get that capability to the warfighter. 

So, I hope that in a brief 

Mr. Conaway. Well, I would be 

Mr. Ahern [continuing]. Brief, why it is the MDD is important 
to us, and what we do in that MDD and how we go forward with 

it. 

Mr. Conaway. A classic example of not answering my question. 

I was more focused on the cost estimate and a rigorous review 
of that, wherever it fell in the system. 

Mr. Ahern. Okay. 

Mr. Conaway. I am not sure it was before we decided to go for- 
ward we decide how much it is going to cost, or after we are into 
it and we decide it is going to cost. 

Mr. Ahern. Can I answer that? 

Mr. Conaway. But 

Mr. Ahern. I am sorry. 

Mr. Conaway. Yes, but I want to make sure that whoever is re- 
sponsible for doing the cost estimate — I come from an oil and gas 
background. And when you get a geologist who has put together a 
new prospect, they get in love with it. And they want it sold, they 
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want it done. The problem is, they have got to go sell it to a third 
party, who takes a different look at it than the guy putting it to- 
gether. 

So, I do not want our folks so in love with their prospect — and 
in this sense, it was an acquisition system — that they lose sight 
of — lose objectivity. 

Is there in that, somewhere we snap the chalk line, an inde- 
pendent — not necessarily the GAO — but an independent review of 
this whole system to that point, so we make sure we do not have 
folks who have fallen in love with a system and are no longer objec- 
tive on the costs and all these other things you mentioned? 

Mr. Ahern. Yes, sir, there is, absolutely. There is an inde- 
pendent cost analysis done on all the major systems by the CAIG, 
the Cost Analysis Improvement Group, on every system. 

So, when we come to snapping the chalk line at what we call 
milestone B, the beginning of EMD, we will have two separate esti- 
mates to look at, one from the service and one an independent cost 
estimate 

Mr. Conaway. Okay. One quick follow up. 

Mr. Ahern [continuing]. Done totally differently. 

Mr. Conaway. The 5000.2 regulation from last August, are you 
going back and applying those to all existing systems? Or is that 
just for new systems going forward? 

Mr. Ahern. It will apply to existing systems, depending upon 
where they are, sir. 

One of the programs that is already in production, it probably 
will not see too many changes, based on the new 5000.2. 

Mr. Conaway. Even if 

Mr. Ahern. But if a program has just started, we will absolutely 
start it in the last six months or so. 

Mr. Conaway. But if changes could rein in some cost overruns, 
though, you would do that, wouldn’t you? 

Mr. Ahern. No, sir. We would not force anything in. We have not 
fundamentally changed the sequence of events that a pro^am goes 
through. What we have tried to do is increase the discipline in fol- 
lowing that sequence of events. 

As Mr. Sullivan remarked, there is room to tailor — or as he said, 
“wiggle” — inside the 5000.2. What we are trying to do is reduce the 
wiggle and ensure that we have a disciplined process that we fol- 
low for each and every of the acquisitions. 

So, I do not think we would — we are conscious of what you are 
saying, sir, and we would not drive cost into a program to adhere 
to the 5000. 

Mr. Conaway. You would drop cost out of it, though? 

Mr. Ahern. I think we will. I am really keen on the idea of doing 
that technology development phase after we do the Analysis of Al- 
ternatives, to go to that phase where we will look at competitive 
prototyping — and we have already done a couple of programs along 
those lines — where we have two or three competitors putting to- 
gether either an entire prototype or key elements of the system, 
and demonstrating it. 

And that, as I said in my remarks this morning, that helps us 
to understand, is the technology available. And frankly, in putting 
together a prototype, it can give us a real leg up on cost estimating. 
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So, I think that this new 5000.2, with its emphasis on that 
phase, should, in fact, help us to drive costs out of programs. It will 
tell us about technology maturity, and it will tell us what is doable 
within a period of time. 

Mr. Conaway. Thank you, Mr. Chairman. 

Mr. Andrews. Thank you, Mr. Conaway. 

Mr. Cooper. 

Mr. Cooper. Thank you, Mr. Chairman. 

Thanks to the witnesses. 

It seems to me that, if there has ever been an alphabet soup of 
bureaucratic quagmire, this is it, with 130-some attempts to reform 
the system since World War II. And I am not sure that anyone can 
even understand whether any of these attempts or reforms worked 
or not. So, I guess our first job is to not make the problem worse. 

When the Secretary of Defense told us at breakfast last week 
that there was something like 50,000 private sector contractors 
whose only job is to oversee other private sector contractors on 
things like contract performance and things like that, I think it 
gives us and the public an idea of how monstrously complex this 
whole process has become. 

It seems to me that — and forgive me, because you gentlemen 
have spent decades studying this, and we are largely new to the 
complexities of this topic — that some of these problems are self-in- 
flicted wounds. 

Mr. Assad talked to us recently and said basically that it is the 
CAIG that comes up with much more realistic cost estimates. But 
sometimes politicians and contractors refuse to listen to those, and 
we prefer the lower numbers, however unrealistic they are. 

So, when it comes to setting and enforcing baselines, well, we 
sometimes enjoy self-delusion, because, guess what, the numbers 
usually, almost always turn out to be higher. 

I wonder about things like the rapid acquisition process, if it is 
essential. And maybe this is just for smaller systems. But if we can 
somehow bypass our own bureaucracy when we need to, why don’t 
we do it more often? 

I also wonder about if there is any good news in here? Is there 
a pony here somewhere? Are there certain systems that are so as- 
tonishingly reliable or productive or necessary for the warfighter? 
You know, have any contractors ever been rewarded for those? 

In my area, we fly still a lot of C-130s, most of which were built 
before I was born. And they are still going, and they are still the 
warhorse. They are still reliable. 

So, I would like to leaven the bad news with the good, if there 
is any good news. And I still want to be reminded that we spend 
more on our defense than almost every other nation on Earth com- 
bined. 

So, we are the policemen of the world. We are the warfighters 
of the world. You know, the value for the taxpayer is increasingly 
essential as our taxpayers are losing patience with lots of different 
things that we are undertaking. 

So, forgive me for the general sort of take on this. But when gen- 
tlemen like you come to us and say there are strategic and tactical 
failures in the procurement of our essential weapons systems, that 
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is from top to bottom. That is the military. That is the Secretary 
of Defense (SECDEF). That is us, the White House. 

So, this may be a task far larger than a simple panel can under- 
take, but we appreciate your guidance. And if, in the time remain- 
ing, if either of you would care to reflect, I would appreciate it. 

Mr. Sullivan. Well, you know, I would like to go back to, after 
hearing you speak there, the chairman asked the question to me, 
what are the things that should change. And I do not know that 
I answered it real well. 

It is those prerequisites, though, that I talked about earlier. And 
when you look at what you just described, we kind of describe that 
as — I do not want to use the word “failures” — ^but a lack of success, 
if you will, at a strategic level, and then down into the execution 
phases. 

And at a strategic level, if you want something good to happen, 
probably the first thing, the first prerequisite is to have less pro- 
grams vying for this money that is available. When you have the 
number of systems — and one of the reasons there are 96 systems 
that are in the portfolio now, is because it is a relatively service- 
centric kind of a system, so you have a lot of parochialism. 

And I think sometimes where you could have a joint solution, or 
you could have a solution that does not have to become an acquisi- 
tion program, the services kind of compete with each other to get 
programs started. 

So, at a top level. Office of the Secretary of Defense (OSD), the 
Secretary of Defense can do a better job of getting a handle on that, 
trying to control the service-centric aspects of this and try to re- 
duce the number of programs that are really competing in 
unhealthy ways for that limited dollar that is out there. 

Mr. Cooper. The average tenure of a SECDEF has been 16 
months over the last 40 years. 

Mr. Sullivan. And in fact, the average tenure — we looked it 
up — of the Under Secretary of defense for acquisitions is around 16 
to 18 months. So, there is no real good continuity. That is an excel- 
lent point. 

I do not know what the answer to that is, unless there is an un- 
dersecretary position that can somehow have a time term, or some- 
thing like that, that would be able to stay in place longer. 

But the turnover really affects a lot of this. You cannot prioritize 
properly. 

Once you cannot kind of keep control of the number of programs 
that are beginning, when you get too many programs into this port- 
folio, you get unrealistic baselines as a result. And I would say, you 
know, that is the other thing, that you need to have requirements 
that are analyzed a lot more. 

I think the key thing that the new 5000 policy does is the pre- 
liminary design review that they are calling for now. They want to 
do that right around the milestone B, which is where they snap the 
chalk line, as Mr. Ahern — that is when they start spending the big 
money. The earlier they do that, the more realistic estimates they 
will get, if they are doing that properly. It is a lot of systems engi- 
neering that has to be done early. 
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And that kind of sorts out, you know, risky technologies. And un- 
realistic requirements are going to drive unrealistic cost and sched- 
ule estimates. 

If you do not understand the requirements that the user is com- 
ing up with, and you do not have discipline there to say we cannot 
do all that right now — you know, there are technologies that you 
have to develop; we can get that in the next generation, but not 
now — these programs will be hard to control. 

Mr. Cooper. Thank you, Mr. Chairman. 

Mr. Andrews. Thank the gentleman from Tennessee. 

The gentleman from Colorado, Mr. Coffman, is recognized for 5 
minutes. 

Mr. Coffman. Thank you, Mr. Chairman. 

Mr. Sullivan, can you give me an example — I think you men- 
tioned immature technologies — can you give me an example of 
that? 

Mr. Sullivan. Well, for example, programs. You know, I hate to 
single out programs, but I will take one that is almost done, I 
guess. The F-22, for example, had technologies that were part of 
the key performance parameters of the aircraft that were very im- 
mature when they started milestone B and started spending the 
big acquisition dollars. 

And they fused avionics on that aircraft. They did not under- 
stand those technologies well at all. In fact, some people would say 
they were not invented when they opened up the factories to build 
that, some of the propulsion technologies. Some of the stealth tech- 
nologies they were not real sure about on that one. 

I do not want to single out the F-22. You can pick almost any 
major — even the C-17, which was a relative — you know, it was a 
big cargo aircraft with relatively mundane requirements, not for a 
cargo aircraft, but in general. And they had some technology issues 
on that with some of the material technology they were using that 
caused them a lot of problems. Very immature technologies on that. 

You can name almost any really made — the Future Combat Sys- 
tems, as, you know, we have looked at that and found that — I do 
not know the exact numbers, but it is maybe 50 or 60 different key 
technologies that are supposed to drive that system. Probably the 
majority of those are too immature to be in product development. 

And the way we look at that, that we have something called 
technology readiness levels that, actually, we recommended that 
the department begin using years ago. And the department has 
started using those. 

And, in fact. Director of Defense Research & Engineering 
(DDR&E) does these technology readiness assessments that Mr. 
Ahern referred to, so now they are doing that on every major weap- 
on system acquisition before it begins. They go in and look at those 
and assign technology readiness levels to those programs. 

Some are still beginning with technologies that are too imma- 
ture. But I would say they are getting better at that. 

Mr. Coffman. And you said words to the effect that there are 
some programs that are not good candidates, that the system does 
not necessarily ferret out programs that are the best candidates. 

Mr. Sullivan. You would like examples of those? 

Mr. Coffman. Could you give one example of that? 
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Mr. Sullivan. Well, that was a kind of a general — more of a gen- 
eral statement. 

I think that it would be — if you want to consider a good can- 
didate to start product development, what we think the best prac- 
tice for that would be is a candidate where you understand the re- 
quirements, you have looked at the requirements and determined 
that there are things you can do and things you cannot do, and you 
have gotten ridden of some of the requirements that are not doable. 

You have looked at your funding, and this program fits into a 
funding profile that the department can count on, and you have 
looked at technologies and things like that. And all of these things 
fit. 

I would say that hardly any of the major weapon system acquisi- 
tions are good candidates, according to that criteria. Lately there 
have been a couple, I think, that we are looking at now that we 
think the department is doing good with. 

I do not know. Small diameter bomb is an example of that, where 
they have really looked at those requirements. 

Mr. Coffman. Okay. Let me ask a question of both of you. 

In 1992, I was with the Marine Corps, and came up here on Cap- 
itol Hill and had a meeting with the officer in charge of the Marine 
Corps liaison program. He was a brigadier general whose name es- 
capes me right now, but he said something that I have never for- 
gotten. And I would like if you would both respond to this. 

He said that, kind of almost — I will paraphrase it. We get weap- 
ons systems that we neither want nor need, but are based on a con- 
gressional — but are manufactured in a congressional district whose 
congressman sees it as a jobs program for their district. 

Can you respond to that? 

And that was his statement in 1992. First of all, did you think 
that that statement was reflective of the environment in 1992? And 
is reflective of the environment today? 

Mr. Ahern. No, sir. 

Mr. Coffman. Okay. 

Mr. Ahern. I mean, let me say — I mean, as you read my biog- 
raphy, I was program manager and Program Executive Office 
(PEO) at that period of time. Obviously, I am aware that there 
are — congressmen have industry in their district. 

But I have never felt — never, ever felt — that we were pressured, 
encouraged or in any way directed to do anything that entered into 
that. I honestly think that we made mistakes, that each time we 
started down the road toward a product development, we have 
done it as well as we can. 

We appreciate your interest and support, but I have never felt 
that we have been pressured into doing something for that kind of 
a reason. 

Mr. Coffman. Okay. 

Mr. Andrews. If the gentleman would yield just a moment, I 
would like to piggyback on this question. 

Mr. Ahern, could you say the same thing about a situation where 
there has been a cost overrun in an existing program and there has 
been an effort to limit it or eliminate it, there has not been con- 
gressional pressure to resist that elimination or cutback? 
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Mr. Ahern. Well, as you know from my portfolio, I have been 
through a number of cost growths, because it is a difficult job. And 
again, I can say, yes, sir, never has happened. Never has happened 
to me personally, and I have had some fairly senior jobs in the Pen- 
tagon and in the Navy and in the Air Force. 

No, sir. It has absolutely never happened to me. And to my 
knowledge, the work that we have done in the Pentagon, the re- 
views, the Nunn-McCurdy reviews, I have not heard that at all, sir. 
No, sir. 

Mr. Andrews. Mr. Coffman, thank you. Mr. Ahern, thank you. 

Mr. Coffman. Mr. Chairman, I am wondering if Mr. Sullivan 
might be able to respond. 

Mr. Sullivan. Yes, I think the first part of your question I can 
deal with, and that is a warfighter saying that they did not get 
what they need. The reasons are many-fold. 

But the one that I look at most is, a lot of times a warfighter 
does not get their urgent needs, because we are busy working on 
very highly complex, single-step-to-big-bang capability systems that 
are draining a lot of funds and time and energy, like the F—22 or 
the Joint Strike Fighter or the Future Combat System. 

Or you can go across the board. A lot of these big, you know, 
kind of unachievable requirements programs take so much time 
and so much money, that I think a lot of times the warfighters — 
there are bill-payers out there for these, right? We all understand 
that. 

So, there are smaller programs that have to pay the price when 
an F—22 program begins with an unrealistic cost and schedule esti- 
mate and every year needs to be plused up in order to get the de- 
velopment through. And so, people pay for that. 

And I think that is something that needs to be looked at, as I 
think the warfighter suffers that way, because these big programs 
are taking the dollars. 

The other thing I would say is, if you look at the Mine Resistant 
Ambush Protected (MRAP) vehicle acquisition, the MRAP acquisi- 
tion actually was pretty good, you know. Once they focused on the 
fact that they needed to counter this threat, and it was an urgent 
threat, they moved very quickly to get MRAPs to the field. 

Before that, I think that this acquisition process hindered the 
ability to do that, because there were other acquisition programs — 
you know, the Army, for example, had other programs that it need- 
ed to fund, and things like that. I think it took a long time for peo- 
ple to accept the fact that the MRAP had — we had to put money 
into this, we had to go with requirements that were doable right 
now, and we had to meet an urgent need. 

Once those decisions were made, that went pretty well. 

Mr. Andrews. If I may, just also, I think one of the reasons it 
received such a high level of congressional scrutiny on an almost 
daily basis. Chairman Hunter at the time, Mr. Skelton watched the 
process like a hawk, as did several other members. And I think 
there is some subjective value — some suggestive value in that. 

Mr. Ellsworth is recognized. 

Mr. Ellsworth. Thank you, Mr. Chairman. And I think my com- 
ments to be more general. 
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First, in fairness, I want to say that we brought up U. Conn.’s 
basketball program. I thought I had better bring up Mr. Sullivan’s 
alma mater 

Mr. Sullivan. Maybe not right now. [Laughter.] 

Mr. Ellsworth. They have been known to throw a few balls 
through the hoop over the years, so hopefully they will get back to 
that. 

Mr. Sullivan. Thank you, sir. 

Mr. Ellsworth. When I first got put on this panel and you dis- 
tributed some paperwork, one of the charts that jumped off the 
screen was the actual chart that showed the acquisition process. It 
reminded me of a Dr. Seuss configuration. And that was funny in 
the Dr. Seuss books, but in military acquisitions, I did not find 
much humor in that, and thought, you know, how in the world can 
anybody navigate their way? 

And as I still try to learn that system, there are just things that 
jump off the page. You would almost have to think it was planned 
confusion. And I hope that is not the case. 

But it would seem to breed the shirking of responsibility, how we 
go through, and how many points we could push off the responsi- 
bility on these decisions. 

And another thing comes to mind, and I think that Mr. Coffman 
said it well. And I heard the president — I think Mr. Cooper and I 
were at a meeting at the White House a few weeks ago where the 
president said, “I have to make decisions on national defense really 
based on national defense, not on a congressman’s district and 
what they make there.” And so, that gives me great hope that 
those decisions will be made on that. 

One of the things we also have to be — and maybe you could talk 
to this — is fairness to the contractor. You know, when we are build- 
ing the hull of a ship and signing that contract, and then it is de- 
signed as we go, as we build up from the basic platform up, not 
unlike the Congressional Visitors Center, where it started off at 
$300 million and kept changing and changing and changing, and 
then went to $600 million. I think it is the same with the heli- 
copter, if we keep adding things as we go. 

We have also got to be able to give these — nobody is going to go 
out and just build a ship and hope that the United States govern- 
ment buys that ship, or get ready and build a factory that is ready 
to go, whether it is two submarines a year, one submarine a year 
and say, hey, I hope they up that someday. 

We have to give the contractor some kind of vision of what we 
are looking at. I know that is tough in two-year cycles and six-year 
cycles and every four years. But I think we have to give them 
something to look at when they are putting employees together and 
buying equipment, and how we address that along with rapid ac- 
quisition. 

Where is the middle ground there between the normal acquisi- 
tion process and rapid? Is there something in between there that 
works better, simplifying it, making the steps down? 

So I guess, if I had direct questions, it would be, you know, 
maybe we can look at the percentage of programs pulled just be- 
cause of cost. The warfighter needed it, but it was just too expen- 
sive. 



21 


Or the percent pulled because we thought we needed it. It sound- 
ed good. It would be nice to have. But then we realize halfway in, 
maybe we do not. It was a great wish, but we did not really want 
it. 

And I am just talking in general. I know that this is our first 
meeting, Mr. Chairman, and I appreciate it, as we try to get our 
arms around it. But maybe you could address the fairness to the 
contractor and how we look to improve that system also. 

Mr. Sullivan. The way I look at that is — that is a really good 
question, because obviously, we are in a — the defense industry is 
an industry that does not necessarily build things in volume, be- 
cause — you were kind of iterating to that. 

There are not real commercial markets for these things. These 
are going to be — you know, you build a couple of submarines and 
you are done. You build a number of fighter aircraft, and there is 
no more market for that. There is no re-sale. There is nothing like 
that. 

So, you are starting out with an industry that — this is very cap- 
ital-intensive, very labor-intensive and low volume. And in addition 
to that, it is cutting edge. 

And I do not think any of us will ever get to a time where it is 
not risky to build defense acquisitions, these big weapon systems, 
because we want them. We want the best in the world. The risk 
could be a lot less than that. 

So, what the department usually does with these big contractors, 
the way they deal with that, I think, is with these cost-plus con- 
tracts that are a necessary part of doing business, because of what 
you referred to. 

No company is going to, on their own nickel, begin to invest in 
the facility and the tooling that they need to take care of the gov- 
ernment, because that is the only game in town. So, if they do not 
get it, they have wasted their money. 

So, the government pays for all of that. So, that is kind of the 
fairness to the contractor, I guess, is we kind of take on their risk. 

On the other hand, we look at the funding on an annual basis, 
and never really give contractors on some programs the stability 
that they need, the security they need in receiving those funds on 
a year-to-year basis to be able to do their job better. 

And when you combine that with the requirements that we have 
already discussed, when you have a contractor working on a weap- 
on system that has capabilities that they really are not able to 
build right away, and they have this cost-plus contract, and they 
have the President and the Congress every year looking at their 
budget, it does create a really unstable environment for them to op- 
erate. So, those are some things. 

Requirements are unachievable. Cost-plus contracting comes up 
a lot. I think a lot of people talk about getting rid of it. I do not 
think necessarily reasonable people think that is a good idea. But 
if you can get to a position where you build — where you have re- 
quirements to build things that are doable, the cost-plus contracts 
make a lot more sense. 

Mr. Ellsworth. Thank you both very much. 

Thank you, Mr. Chairman. 

Mr. Andrews. Thank you, Mr. Ellsworth. 
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Mr. Hunter is recognized for 5 minutes. 

Mr. Hunter. Thank you, Mr. Chairman. 

And in the interest of time here, we will start off with my first 
question. In 2003 to 2008, we have had kind of a permissive envi- 
ronment. I think you would agree to that. 

Who was responsible, if you had to lay it out in terms of who was 
most responsible to least responsible for the permissive environ- 
ment, that maybe the services and Congress and contractors took 
advantage of and are now getting reined in? Who is responsible? 
When it comes to DOD, Congress, contractors, who would you lay 
the blame on the most from 2003 to now? 

Mr. Ahern. I started here in 2006. And, of course, I was an in- 
structor before that. 

I would not go to blame. I would say that the discipline in look- 
ing at the system, at the elements of a development going forward, 
were not as rigorous then as they are now. But I am not sure that 
I could say that the blame is there. The system was always there. 
The system has always been there. 

And there have been some good programs started, and they are 
continuing to execute through that period of time. I think of the P- 
8, the Navy’s replacement for the P-3, which I think was started 
in that period of time. 

So, I am unable to ascribe blame. I think a number of the pro- 
grams that are in the portfolio Mr. Sullivan mentions are, in fact, 
executing over the last 5 years quite well. I believe that that is in 
his report. 

So, I am not sure that there was a permissive environment in 
that period of time. I think that what we have done recently, or 
in the new 5000, is to add — and in other things that we have 
done — to add additional expectations. 

But I would not agree that there was a permissive environment 
during that period of time. And I think, as I said, there are a num- 
ber of programs that have not shown cost growth, that were started 
in that period of time. 

Mr. Hunter. Almost every service acquisition officer who I have 
talked to in the last — I have only been here for about 2.5 months. 
But everyone that I have talked to said there was indeed a permis- 
sive environment, and that contractors took advantage of it — not 
necessarily at the big level programs, but down at the lower levels, 
middle levels, and that that is being reined in now. 

And industry is suffering to some extent. The services are having 
to be more responsible, and DOD is having to be more responsible, 
too. 

Mr. Sullivan. One of the comments I would make is that, you 
know, one of the things that started around, in the 2000 timeframe 
is when we started having these acquisition programs like Missile 
Defense Agency (MDA), Future Combat Systems, Joint Strike 
Fighter. Some of these very, very complex what they call systems 
of systems, as opposed to just building a weapon system. You had 
the Future Combat System, which is a — ostensibly, there are 19 
programs kind of working within that that somebody is trying to 
coordinate. 

Joint Strike Fighter is three separate fighter aircraft variants 
that Lockheed Martin is trying to do at one time. There are a lot 
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of programs out there like that. So, I do not think the government 
particularly did that very well. 

The MDA is an acquisition program. It is an agency. And it 
has — I do not know the number now, but it is in the teens of highly 
complex programs that are supposed to work together. 

So, I think one of the things that happened is that it got much 
more complex, so complex that we cannot understand it too well. 

And I think the permissiveness, a lot of that might have come 
from what happened. You know, there was an attempt at acquisi- 
tion reform in the mid 1990s. 

Mr. Hunter. Well, as complexity goes up, costs go up and every- 
thing else goes up. 

Mr. Sullivan. I think absolutely, yes. 

Mr. Hunter. Let me jump in here, because we have to run. 

How do you — and this kind of goes to those mid-level warfighter 
programs that guys really need, programs like Joint Improvised 
Explosive Defeat Organization (JIEDDO), Task Force (Odin) and 
things that we funded, because of immediate current threats that 
the warfighter has begged for, but that Congress almost has to 
force on the different services. And these programs start off agile 
and responsive, and they end up slow, unresponsive, bureaucratic. 

And these are all programs where the ongoing operation is just 
as important as the initial acquisition of them. 

And it seems like everything works well when it first starts off. 
It is lean and mean, and then it kind of gets out of control, once 
they get their billions and they hire 300 bureaucrats. Everything 
slows down and they become unresponsive, and they do not nec- 
essarily do what they were intended to do in the first place. 

But that is more of an ongoing thing, but it still has to do with 
the initial acquisition and programs not doing what they are sup- 
posed to do after the initial acquisition. 

How do you fix that? 

Mr. Ahern. Well, there are a couple of things that occur to me 
immediately, congressman. One is the MDA that Mr. Sullivan men- 
tioned a couple of minutes ago. That was initiated, I think, in 2002 
timeframe, and is one that had a very near-term expectation that 
they would develop a capability by — develop and field a capa- 
bility — by 2004. 

And then, as time went — and then continue on. And I think the 
MDA has a number of elements in development moving toward a 
block capability that will enable them to expand upon that initial 
capability. 

And what we have done with the MDA is to ensure that they 
continue to have that Research, Development, Test & Evaluation 
(RDT&E), or that technology development focus, and move toward 
transitioning the capabilities that they are developing to the serv- 
ices to operate as rapidly as we can. It requires focus in that area. 

Another one that comes to my mind that we are working on now 
is the Intelligence Surveillance and Reconnaissance (ISR) task 
force that Secretary Gates started. And that is one that I support 
from an acquisitions standpoint. And it is absolutely to push ISR 
resources into the operating areas as quickly as we can. 
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I expect it to operate for a period of time. I expect to continue 
to push on it. And the same kind of thing is happening with the 
MRAP. 

And it really is, as — it is up to the people, the discipline that I 
mentioned earlier on, to have, continue to have that interest in get- 
ting the joh done to support the warfighter. And I cannot say it any 
better than that. 

We have to do our jobs responsibly to ensure that we do main- 
tain the focus on the urgent operational needs, the support for the 
warfighters, as I think we are doing in the ISR task force, as we 
did with the MRAP and continue to support the MRAP, and as we 
are doing with the MDA. Those are three examples that occur to 
me. 

Mr. Andrews. Thank you, Mr. Hunter. 

The chair recognizes Mr. Sestak for 5 minutes. 

Mr. Sestak. Thanks, Mr. Chairman. 

I honestly believe that the requirement side, particularly since 
General Cartwright used to have J8, really has come a long way 
from the Cost Benefit Analysis (CBAs), and the modeling he has 
done down there. So, I want to focus more on the acquisition side. 

But could I ask you, sir, if you could get back to us? My limited 
understanding is that under Instruction 5000, all the modeling that 
is used on the requirement side has to go through a validation, 
verification and analysis by an independent somebody to make sure 
the models used are well. My understanding is, probably only 5 
percent have. 

Could you get back to us on that 

[The information referred to can be found in the Appendix on 
page 80.] 

Mr. Ahern. Yes, sir. 

Mr. Sestak [continuing]. Because we are using models in the re- 
quirement side that have never been validated. How good are they? 

So, I think that is what I would really like to know. My under- 
standing is, even though there is a requirement to in 5000, hardly 
any of them have. And I think that is something that does not bode 
for credibility on the modeling being used. 

The other side of the acquisition side, I was quite taken by your 
comments about the Earned Value Management (EVM) program, 
where we are supposed to forecast final cost. Also, sir, with you, 
I was taken with how you want to try to establish prerequisite in- 
dicators to try to get realistic costs, to kind of break this tyranny 
of optimism we have. 

The other day, when the Littoral Combat Ship (LCS) came for- 
ward, initially it was supposed to be $250 million. Now it is $460 
million. The internal DOD figures only have a 50 percent con- 
fidence factor that that is going to be the actual cost. 

I guess my point is, we have Nunn-McCurdy. Thirty programs 
the last three years have come over to us, breaking, breeching 
Nunn-McCurdy. Nice monitoring system, but no enforcement, no 
teeth on it. 

You mentioned the CAIG, sir. CAIG estimates are not manda- 
tory, yet they turn out to be, by and large, much more right than 
the services. Should we. A, make the CAIG estimates mandatory? 
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Number two, shouldn’t the Congress have exposure to those con- 
fidence factors that you have internally to the services, for exam- 
ple? The aircraft carrier that is going to be built here in a few 
years is going to cost, according to some estimates, $13.5 billion. 
But the internal confidence factor of costing on that is 37 percent. 

We are building a lot of Virginia class submarines. But the inter- 
nal confidence factors on the costing for the Virginia class sub- 
marine — two of them being built here in the next years — is less 
than 50 percent. 

Should Congress here have access to that data, so before we com- 
mit to something like an LCS, we know at that time maybe that 
20 percent confidence factor before we get too far down the road? 

Sir. 

Mr. Ahern. Sir, taking it in terms of the CAIG being mandatory, 
I think we need to maintain that balance. The experience that I 
have is, we tend to use the CAIG estimate for the estimate for the 
program. But we need to have that dialogue between the CAIG and 
the service estimate, the program office estimate, because they are 
different. They are done differently. 

And I think that that dialogue is necessary. And if we made the 
CAIG mandatory, I think that the service cost estimating, which 
enables that dialogue, would 

Mr. Sestak. Do you think we should have access to both cost 

Mr. Ahern. Well, I think you should have access to the cost esti- 
mates. 

Mr. Sestak. I mean, if they are not an 

Mr. Ahern. I am not sure 

Mr. Sestak [continuing]. Independent one, like you said, 
shouldn’t Congress know before we dole the national treasure out, 
what CAIG’s estimate is versus the service’s, or at least the con- 
fidence factor that is coming forward? 

Mr. Ahern. I think we do send over the — I mean, the results of 
that estimating is done in the Acquisition Program Baseline (APB) 
that is reported in the Selected Acquisition Report (SAR) reports 
for the programs. 

So, I think that the results of that work is absolutely reported 
to you in the SARs that come over on all the major programs when 
they initiate them at a program milestone. I really do think it is 
there. 

Mr. Sestak. All right. I probably missed it. 

May I ask another question? I think my time — oh, I have time 
for one more. 

My question is on the more jointness. I was always quite taken 
with what Representative Skelton and many did with the JCS, you 
know, with having — back in the Goldwater-Nichols day, when the 
chairman walked in after that was passed, everybody else stood up. 
Before that, the JCS would not stand up when he walked in the 
room, because he was one of equals. 

To some degree, do you think, as we have a wonderful system, 
including having OSD involved in the JCIDS process now, which 
it was not before — I mean, a few years ago they were brought in — 
do you think that we need to structure the JROCs, but not so much 
as what we did to the JCS, so it is not everybody is equal there? 
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You may get the least common denominator. It is my program 
versus your program. 

But do we need a Goldwater II, in a sense, to say there should 
he one final requirements guy before it goes any further, rather 
than least common denominator. Any comments? 

Mr. Sullivan. Well, my comment on that is that, you know, 
Goldwater-Nichols was to make jointness. And it did it on the oper- 
ations side, I think. You know, if we see the way 

Mr. Sestak. But not the procurement side. 

Mr. Sullivan. Not the procurement side. So, there, I think you 
raise a really excellent point. 

These programs have to he acquired jointly, and they are not 
right now. There is still too much service-centricity, even in the 
JCIDS process, which, when you read the JCIDS policy, it is really 
pretty good policy. 

Mr. Sestak. It is. 

Mr. Sullivan. It establishes functional capability boards and a 
lot of joint matrix processes that are supposed to look at require- 
ments and weigh in in a purple-suited way, if you will. 

Those things are not in effect right now. They have not staffed 
them properly. 

I think you still — I do not think that the policy has been imple- 
mented well. 

Mr. Sestak. I am out of time. But my only thing is, when it fi- 
nally gets to the JROC, it is wonderful work up till there. But that 
final decision, I just do not know if you need to change it like the 
JCS 

Mr. Sullivan. Yes, if I could just comment briefly on one other 
thing you were talking about, the CAIG estimates. 

We have done a lot of work in that area. And we found that the 
CAIG usually has a little bit better accurate estimate, but still, far 
from what the outcomes usually tend to be. And the department 
does not always accept those estimates. 

And I think that an independent CAIG would be something that 
might be helpful to the Congress, something similar to how they 
do Director of Operational Test & Evaluation (DOT&E). They es- 
tablished an independent director. I think that is something that 
might be helpful. 

Mr. Ahern. I would like to go back and comment on Mr. Sulli- 
van’s last comment. I think that the more we emphasize that need 
to do the work prior to beginning a program, the product develop- 
ment, we will have a far better informed, independent as a well as 
a service estimate. 

So, I do not think the answer, sir, is structurally to set up an 
independent CAIG or to do something along those lines. I think it 
is to ensure that we have more information on the products before 
we get into the product development. 

The competitive prototyping, the insistence upon technology ma- 
turity are the kinds of information that we need to have. Otherwise 
we are using old information, parametrics. Well, we did it this way 
a couple of years ago, so we will use that kind of way. Or circuit 
boards cost $1.98 in 2005, so assume they cost $1, you know, $2.05. 
We need to have better information than that. 
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That is why I think, yes, the CAIG estimate, the service esti- 
mates are critically important. But to improve them, what we need 
to do is improve the information that goes into them. 

Mr. Sullivan. But in order to improve that information, you 
might need an independent assessor overseeing all that 

Mr. Sestak. You could be joined and do both. 

Mr. Sullivan. This could be a chicken and egg thing. I do not 
know. 

Mr. Andrews. I would like to thank Mr. Sestak, particularly for 
the observation about the dichotomy between the jointness in oper- 
ations — 

Mr. Sullivan. Yes. 

Mr. Andrews [continuing]. But the lack of jointness in procure- 
ment. I think it is a very critical point 

Mr. Sullivan. Yes, very 

Mr. Andrews [continuing]. That goes to a lot of what Mr. Ahern 
and Mr. Sullivan said. Thank you. 

I am going to ask if Mr. Conaway has any concluding remarks. 

Let me also mention that, if any member of the panel would like 
further analysis or questions, please submit them in writing. I am 
sure the witnesses would comply with a written answer. 

Mr. Conaway. 

Mr. Conaway. Well, witnesses, thank you very much for coming 
this morning at an unusual hour for a hearing like this. I appre- 
ciate that. 

I would like to follow up with you with respect to these CAIG es- 
timates and the ones that are actually used, and what requirement 
there is for reconciling the differences between the two on the front 
end. 

Mr. Conaway. And then, Mr. Sullivan, you may have some his- 
torical data about tracking reasonable estimates, CAIG estimates 
and reality. 

Mr. Sullivan. Yes, I do. 

[The information referred to can be found in the Appendix begin- 
ning on page 78.] 

Mr. Conaway. It might be helpful for us to look at it. 

But in closing, it is actually the panel’s responsibility to find leg- 
islative issues that need to be addressed on a go-forward basis. So, 
if there is legislation that you see is needed to help what we are 
trying to do and trying to accomplish, which I think all of us have 
the exact same goal, please point that out to us, as well, because 
that is really the goal of what we are trying to get done. 

Chairman, thank you. 

Mr. Andrews. Thank you. 

I would like to thank the witnesses for their outstanding prepa- 
ration. We are going to call on you as the process goes forward, I 
am sure. 

There are two items I would ask you to supplement the record 
with. 

Mr. Ahern, I am interested in your views on how we might elimi- 
nate the wiggle room that has been identified in the 5000 guidance. 
I know you are working very avidly on that. We would be inter- 
ested in your direction to us on how that process can be sharpened 
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and improved. I thought your comments this morning were very 
edifying. 

[The information referred to can be found in the Appendix on 
page 77.] 

Mr. Sullivan, I would like your views on the question of the 
SARs. The selective acquisition reports that come over are an excel- 
lent tool for the Congress to evaluate problems. As I understand 
it — and correct me if I am wrong — I think all the SARs come post- 
milestone B. Am I right about that? 

Mr. Sullivan. Yes. 

Mr. Andrews. Yes. I am interested in whether there is a similar 
analytical tool that could be created by legislation pre-milestone B, 
so that we could get an early warning signal that there is some- 
thing wrong in the design phase. 

Now, I understand it is a whole different set of questions, but I 
think that a SAR-type tool pre-milestone B would be very useful 
for us. I would be interested in your thoughts. 

And you, as well, Mr. Ahern. 

[The information referred to can be found in the Appendix begin- 
ning on page 77.] 

Mr. Ahern. All right. 

Mr. Andrews. The committee is going to — the panel is going to 
proceed after we return from the recess with another hearing. We 
will be consulting with the minority as to what the topic ought to 
be with respect to that. 

I am confident that we are going to broaden the issue of defining 
what I began this morning talking about, the delta, the gap be- 
tween what we are paying for and what we are getting, beyond the 
issue of the major weapons systems, as well. I think we have had 
an excellent discussion of that subject, but we want to go beyond 
that to the budget, the huge — the majority of the procurement 
budget that is not major weapons systems. 

You have given us some very sobering news this morning, nearly 
$300 billion in overruns. And I think the good news we hear is that 
there are tools in place for us to understand the causes of these 
problems, which then gives us the ability to find solutions. 

What is disturbing, of course, is that the trending is in the wrong 
direction. If you look at the difference between 2003 and 2008, I 
think your data are accurate. The problem is getting worse and not 
better. And I think that goes to Mr. Conaway’s point, and Mr. Coo- 
per’s point earlier. You know, the ream upon ream of assessment 
of this has gotten us nowhere — worse than nowhere. 

So, we are interested in trying to put teeth into the decision- 
making process in a way that makes this work. 

Again, the record will be open for any member to submit more 
questions to the panel. 

We thank both panelists for excellent presentations this morning, 
and we stand adjourned. 

[Whereupon, at 8:54 a.m., the panel was adjourned.] 
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Prepared Remarks of Chairman Rob Andrews 
Panel on Defense Acquisition Refomi 

Hearing on Measuring Value and Efficiency: How to Assess the Perfonnance of the Defense 
Acquisition System 
April 1, 2009 

Welcome everyone to today's hearing on measuring the performance of the defense 
acquisition system. Our witnesses today are Mr. David G. Ahern, the Director of Portfolio 
Systems Acquisition in the Office of the Under Secretary of Defense for Acquisition, 
Technology, and Logistics, and Mr. Michael J. Sullivan, Director for Acquisitions and Sourcing 
Management at the Government Accountability Office. 

Today is the panel's first hearing, and we expect it to be the first of many as we proceed 
to examine the questions outlined in the committee’s work plan. Our first question is a 
fundamental one: how' do wc measure how well the defense acquisition system is performing? 
We ail have our own individual judgments of this question, and it would be unlikely that the 
panel would exist if not for a general understanding that the system often does not perform well. 
But I think it is nonetheless essential to determine whether there are solid, objective measures of 
to what extent and how we believe the system may be failing. 

We oflcn criticize the executive branch, regardless of the issue or the agency, when we 
believe that it has not set dear priorities or established clear benchmarks and metrics to measure 
pertbnnance. I believe we should live by the same standard. It is my hope that as part of the 
panel’s work we can measure the shortcomings in the defense acquisition system or at least put 
the Department of Defense on a path to establish its own measure. Em not sure we will know 
the answer to this question at the end of today's hearing, but hopefully we'll have made a lot of 
progress. I thank the witnesses in advance for their excellent testimony and for their willingness 
to join us at this rather early hour. 

I now turn to Mr, Conaway for any remarks he would like to make. 
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Statement of Ranking Member Conaway 
Hearing of the 

Defense Acquisition Reform Panel 
on 

Measuring Value and Efficiency: How to Assess the 
Performance of the Defense Acquisition System 


April 1, 2009 


Good morning, Mr. Chairman, ladies and gentlemen. I think it's 
appropriate for the Panel's first hearing that we have a senior member 
of the defense acquisition system and GAO's senior acquisition 
management professional sitting side by side. It's not often that 
members of this committee get a chance to talk to the Department and 
GAO at the same time. Thank you, gentlemen, for making this possible 
and agreeing to be with us at such an early hour. 

The first question this panel identified as part of its work plan was 
whether there is a method to reasonably measure the ability of the 


defense acquisition system to deliver the goods and services needed by 
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the warfighter, to do so in a timely fashion, and to do so at a fair price. 
Today's hearing will likely not answer the larger philosophical question 
about how one measures value in defense acquisition, but it is an 
important first step for us to understand how DoD and GAO currently 
assess performance in one segment of defense acquisition - the major 
weapons systems programs that were a focus of the assessment GAO 
released this week. 

These programs receive a great deal of scrutiny by Congress and 
by the media for good reason. GAO's report reveals that nearly 70 
percent of DoD's 96 largest weapons programs were over budget $296 
billion, or nearly 42 percent. This is simply unacceptable. Everybody 
understands why we cannot continue to tolerate these cost increases - 
there is little more to be said on the subject. But what we don't hear as 
much about, is that GAO had encouraging words to say about steps the 
Pentagon has taken to improve acquisition outcomes, including early 
stage systems engineering, prototyping, measurable yearly plans. 
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increasing accountability, and minimizing requirements creep. The 
report states, "These changes are consistent with the knowledge-based 
approach to weapons development that we have recommended in our 
work. If implemented, these changes can help programs to replace risk 
with knowledge, thereby increasing the chances of developing weapon 
systems within cost and schedule targets while meeting user 
needs. ..These are encouraging signs, but to improve outcomes on the 
whole, DoD must ensure that these policy changes are consistently 
implemented and reflected in decisions on individual programs." I 
hope we hear more today about these positive improvements that DoD 
is making, and what more needs to be done. 

Of course, we are likely to learn that much of what DoD does to 
measure performance is already statutorily required. I also hope our 
witnesses feel free to share their views on laws and regulations that are 


not assisting in their efforts to obtain the best value and capability for 
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our warfighters. There is a balance to be struck between setting high 
expectations and over regulating the system. 

With that I will conclude and again thank my fellow members and 
you, Mr. Chairman. I look forward to the witnesses' testimony. 
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Measuring Value and Efficiency: How to Assess the Performance of the 
Defense Acquisition System 

Mr. David G. Ahern 
Director, Portfolio Systems Acquisition 
Office of the Under Secretary of Defense 
(Acquisition, Technology, and Logistics) 

Good morning Chainiiaii Andrews, Congressman Conaway, and Members of the 
Panel. Thank you for the opportunity to appear before you today to discuss how the 
Department of Defense assesses the performance of the Defense Acquisition System. 

Any attempt to measure the return on the taxpayer’s dollar that we arc achieving in 
the acquisition system must be viewed from both strategic and tactical perspectives. In 
doing so, we must be prepared to answer a number of questions. For example: 

a. Has the Department made the optimal investment decisions when deciding 
what weapon system programs to develop and field? 

b. Has the Joint Staff established appropriate requirements through the .loint 
Capabilities Integration and Development System (JCIDS)? 

c. Has the Department fully funded its weapons systems at initiation through the 
Planning, Programming, Budgeting, and Execution System (PPBES)? 

d. Has the Department, using DoD Instruction 5000.2 which guides the 
acquisition process itself, ensured that new programs have a solid foundation in terms of 
technology maturity, cost estimates, systems engineering processes, etc., at the time they 


are initiated? 
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e. Is the acquisition community disciplined in its execution of our programs, 
taking prudent steps to control requirements changes that might drive up cost or delay 
schedules? And are we continuously looking for ways to reduce cost or unnecessary 
requirements both in individual programs and across all our major defense acquisition 
programs? 

f Are wc using all the tools in our toolkit to drive positive outcomes? For 
example are we writing contracts that reward only good performance? Have we 
empowered our acquisition program managers to be proper stewards of taxpayer dollars? 
Are contractors’ earned value management systems collecting the right information at the 
right time to warn of poor execution? Is the acquisition workforce large enough, with the 
proper training and experience, to properly oversee our programs? 

g. For programs that do experience cost growth, how docs the Department decide 
whether to continue, modify, or terminate the program? 

Determining the “Best Value” Acquisition Solution for the Warfighter 

To address the question of whether we are making the right investment decisions, 
the Department uses Capability Portfolio Management (CPM). CPM advises the Deputy 
Secretary and the Heads of the DoD Components on how to optimize capability 
investments across the defense enterprise (both materiel and non-materiel) and minimize 
risk in meeting the Department's capability needs in support of strategy. Under CPM, 
recommendations arc made regarding integration, coordination, and synchronization of 
capability requirements to capability investments. It is used to evaluate capability 
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demand (both warfighting and non-warfighting) against resource constraints, to identify 
and assess risks, and to suggest capability trade-offs. 

The Joint Staff s Joint Capabilities Integration and Development System (JCIDS) 
process was created to support the statutory requirements of the JROC to validate joint 
warfighting requirements. JCIDS, .shown pictorially below, is a key supporting process 
for DOD acquisition and PPBE processes, with the primary objective to ensure the joint 
warfighter receives the capabilities required to successfully execute the missions assigned 
to them. When a Service or agency detennines that they may have an issue, they perform 
a capabilities-based assessment to identify: the capabilities (and operational performance 
criteria) required to successfully execute missions; the shortfalls in existing weapon 
systems to deliver those capabilities and the associated operational risks; and the possible 
non-materiel solutions or the need for materiel solutions for the capability shortfalls. The 
results are documented in an Initial Capabilities Document which, upon approval, 
validates the capability need and supports two alternative processes. First, it can support 
the development and implementation of a non-materiel solution. Second, it can support 
the initiation of the acquisition process, through a Materiel Development Decision, to 
identify and develop a materiel solution. 
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UNCLASSIFIED- 


JCIDS and Acquisition 


^ W -0-[^A H i A r^iAi^s^n.1 

UDD HSA MSB MSC 

g 


CBA = CapaMilies Based Assesarienl 

MOD = Malenel DevstopmenI Bedsiw 

ySA - Waleiiei So!u!k)r>s Analysis 

EiWO - Er'gineenng 5 Manulaclieing Oev^isnent 

ITN’rLASSJFlEO 6 


When the ICD demonstrates the need for a materiel solution, the JROC shall 


recommend that the Milestone Decision Authority (MDA) consider potential materiel 
solutions. The MDA, working with appropriate stakeholders, shall determine whether 


there is sufficient information to proceed with a Materiel Development Decision (MDD). 


The MDD, described below, begins the Materiel Solution Analysis Phase and represents 
the formal entry point into the acquisition process. It is mandatory for all programs. 



Mandatory "Materiel Development Decision” 
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At the MDD review, the Joint Staff shall present the JROC recommendations and 
the DoD Component shall present the ICD including: the preliminary concept of 
operations, a description of the needed capability, the operational risk, and the basis for 
determining that non-materiel approaches will not sufficiently mitigate the capability gap. 

The Director, Program Analysis & Evaluation (DPA&E) proposes study guidance 
for the Analysis of Alternatives (AoA). The AoA focuses on identification and analysis 
of alternatives, measures of effectiveness, cost, schedule, concepts of operations, and 
overall risk. The AoA also assesses the critical technology elements associated with each 
proposed materiel solution, including technology maturity, integration risk, 
manufacturing feasibility, and, where necessary, technology maturation and 
demonstration needs. To achieve the best possible system solution, emphasis is placed on 
innovation and competition, but its important to emphasize that the AoA examines the 
full spectrum of alternatives; starting with current capability and moving to an entirely 
new material solution with the goal of balancing the capability needs, with what the 
Department can effectively acquire and afford to achieve the best value proposition for 
our Nation. 

The Materiel Solution Analysis Phase ends when the AoA has been completed, 
materiel solution options for the capability need identified in the approved ICD have been 
recommended by the lead DoD Component conducting the AoA, and the phase-specific 
entrance criteria for the initial review milestone have been satisfied. After a Milestone A 
review, the program moves to the Technology Development phase. 
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The purpose of the Technology Development phase is to reduce technology risk, 
determine and mature the appropriate set of technologies to be integrated into a full 
system, and to demonstrate critical technology elements on prototypes. I would note that 
if, during Technology Development, the cost estimate upon which a Milestone A 
certification was based increases by 25 percent or more, the MDA must consult with the 
JROC to determine whether the resources required to develop and procure the system 
remains consistent with the priority level assigned by the JROC. If not, the MDA may 
rescind the Milestone A approval. 

During the Technology Development phase, the strategy and associated funding 
provides for two or more competing teams producing prototypes of the system and/or key 
system elements prior to, or through, Milestone B. Prototype systems or appropriate 
component-level prototyping are employed to continue reduction of e technical risk, 
validate designs and cost estimates, evaluate manufacturing processes, and refine 
requirements, again with tlie goal of ensuring the acquisition enterprise pursues the best 
value solution to meet warfighter needs. 

The project exits the Technology Development Pliaso when an affordable program 
or increment of militarily useful capability has been identified; the technology and 
manufacturing processes for that program or increment have been assessed and 
demonstrated in a relevant environment; manufacturing risks have been identified; a 
system or increment can be developed for production within a short timeframe (normally 
less than 5 years for weapon systems); or, when the MDA decides to terminate tlie effort. 
During Technology Development, the user prepares the Capability Development 
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Document (CDD) to support initiation of the acquisition program or evolutionary 
increment, refine the integrated architecture, and clarify how the program will lead to 
joint warfighting capability. The CDD, informed by technology maturity, life cycle cost, 
and schedule considerations, builds on the ICD and provides the detailed operational 
performance parameters necessary to complete design of the proposed system. 

Executing the “Best Value” Program 

The Acquisition Program Baseline is the key document for program management. 
It reflects the approved program being executed. It is the “Baseline Description" of the 
program and includes sufficient parameters to describe the cost estimate (also refeiTed to 
as the “Baseline Estimate" for major defense acquisition programs), schedule, 
performance, supportability, and other relevant factors. The APB becomes the source 
document when determining whether a program has committed a “Nunn-McCurdy" 
breach for cost or schedule. The first APB is approved by the MDA prior to entry into 
EMD. A Milestone B decision follows the completion of Technology Development and, 
where successful, marks the start of the Engineering and Manufacturing Development 
(EMD) phase. 

The purpose of the EMD Phase is to develop a system or an increment of 
capability; complete full system integration (technology risk reduction occurs during 
Technology Development); develop an affordable and executable manufacturing process; 
ensure operational supportability with particular attention to minimizing the logistics 
footprint; implement human systems integration; design for producibility; ensure 
affordability; protect critical program information by implementing appropriate 
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techniques such as anti-tamper; and demonstrate system integration, interoperability, 
safety, and utility. Transition into EMD requires full funding (i.e., inclusion of the dollars 
and manpower needed for all current and future efforts to carry out the acquisition 
strategy in the budget and out-year program), which is programmed in anticipation of the 
Milestone B decision. The CDD, Acquisition Strategy, Systems Engineering Plan, and 
Test and Evaluation Master Plan (TEMP) guide the EMD phase. 

The Acquisition Strategy describes how the PM plans to employ contract 
incentives to achieve required cost, schedule, and performance outcomes. EMD 
effectively integrates the acquisition, engineering, and manufacturing development 
processes with Test and Evaluation (T&E). T&E is conducted in an appropriate 
continuum of live, virtual, and constructive system and operational environments. 
Developmental and operational test activities are integrated and seamless throughout the 
phase. The SEP describes the program's overall technical approach, including key 
technical risks, processes, resources, metrics, and applicable performance incentives. It 
also details the timing, conduct, and success criteria of technical reviews. 

Successful completion of the EMD phase leads to a Milestone C decision and 
entrance into the Production and Deployment Phase. The pui-pose of the Production and 
Deployment Phase is to achieve an operational capability that satisfies mission needs. 
Operational test and evaluation determines the effectiveness and suitability of the system. 
The MDA makes the decision to commit the Department of Defense to production at 
Milestone C. Milestone C authorizes entry into Low I^ate Initial Production (LRIP) for 
major defense acquisition programs, into production or procurement (for non-major 
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systems that do not require LRIP), or into limited deployment in support of operational 
testing for major automated information systems or software-intensive systems with no 
production components. 

LRIP is intended to result in completion of manufacturing development in order to 
ensure adequate and efficient manufacturing capability and to produce the minimum 
quantity necessary to provide production or production-representative articles for 
operational test and evaluation, establish an initial production base for the system; and 
permit an orderly increase in the production rate for the system, sufficient to lead to full- 
rate production upon successful completion of operational (and live-fire, w'here 
applicable) testing. 

Continuation into full-rate production results from a successful Full-Rate 
Production Decision Review by the MDA. This effort delivers the fully funded quantity 
of systems and supporting materiel and services for the program or increment to the 
users. During this effort, units will typically attain Initial Operational Capability (IOC). 

Throughout the acquisition process, the Department uses a variety of program and 
technical reviews to assess program status and for decision making purposes. Among 
these reviews are the Defense Acquisition Board (D.AB) reviews. The DAB advises the 
Under Secretary of Defense for Acquisition, Technology, and Logistics (USD(AT&L)) 
on critical acquisition decisions. TheUSD(AT&L) chairs the DAB. Configuration 
Steering Boards (CSBs) meet at least annually to review all requirements changes and 
any significant technical configuration changes for major defense acquisition programs in 
development that have the potential to result in cost and schedule impacts to the program. 
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Such changes will generally be rejected, deferring them to future blocks or increments. 
Changes are not approved unless funds are identified and schedule impacts mitigated. 
Program Support Reviews are a means to inform an MDA and Program Office of the 
status of technical planning and management processes by identifying cost, schedule, and 
performance risk and recommendations to mitigate those risks. Defense Acquisition 
Executive Summary (DAES) reviews. The DAES provides an early-warning report to 
USD(AT&L) and ASD(NII). The DAES describes actual program problems, warns of 
potential program problems, and describes mitigating actions taken or planned. At 
minimum, the DAES should report program assessments (including interoperability), unit 
costs, and current estimates. 

In addition to program-level assessments, the Department uses quantitative tools 
to assess contract performance. Earned Value Management (EVM) is a widely accepted 
industry best practice for project management that is used not only in DoD, but other 
Federal government agencies as well. It is an integrated management system that 
coordinates the work scope, schedule, and cost goals of a program or contract, and 
objectively measures progress toward these goals. EVM is a tool used by program 
managers to: (1) quantify and measure program/contract performance, (2) provide an 
early warning system for deviation from a baseline, and (3) provide a means to forecast 
final cost and schedule outcomes. 

The overview provided here is focused on major defense acquisition programs and 
reviews conducted by the Office of the Secretary of Defense. However, the Military 
Departments apply the same principles and processes to smaller programs that do not 
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reach the threshold of a major defense acquisition program. In addition, there are similar 
processes for information systems and processes to address unique activities, such as for 
missile defense. In each case, however, analogous processes and procedures provide the 
Department the information needed to decide when and whether to commit resources and 
to oversee program execution. 

The recent issuance of the new DoD Instruction 5000.2 results in the 
implementation of many initiatives aimed at ensuring programs are started with a solid 
foundation, are focused on disciplined execution, and deliver capability to the warfighter 
within cost and schedule parameters. It will take time for us to fully realized the benefit 
of these policy initiatives. However, we will continue to look for opportunities to 
improve the Defense Acquisition System. 1 look forward to forward to answering any 
questions you might have. 
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DEFENSE ACQUISITIONS 

Measuring the Value of DOD’s Weapon Programs 
Requires Starting with Realistic Baselines 


What GAO Found 

GAO employs a set of knowledge' metrics to determine wheth(>r programs 
have attained the right knowledge at c-ritica! points over I tie course of a 
weaf>on system acquisition, ami facilitate (he identificat ion of potential 
problems that could lead to cost, schedule, or peiibrmaiu;e shortfalls. In 
essence, knowledge supplants risk over time. Key knowledge points and 
metrics irndude 1) achieving a high level of technology maturity at the stiirt of 
program development, 2) rcacliing design stability at the sys!em-lc\ el critical 
design rewew, and 3) demonstrating that critictal numufa(.'turing processes arc- 
in ctontrol before starting production. By applying llu-sc metric-s to selected 
progiTims in DOD’s 2008 portfolio of major defense acquisil ions, GAO found 
that most programs have started system devc-lopmcnt without mature 
technologies and moved into system demonstration with low levels of design 
stability. GAO has determined that program.s with immature l.(^chnologics and 
unstable designs ha%'C cxporionced sigiuKc-ant c-ost and schedule growtli. 

Program outcome metrics- -<nuint ita(ivc measures of cost, schedule, and 
perfornuuice over time — provide u-sc-ful indUaitors ol' (he health of actiuisilion 
programs and whether they are meeting their intcnckKl goals. When asscvssec! 
regularly for changes and the rcfisons that cause ciianges, these indi(.-atc.>rs can 
l)(^ valiiatile tools for improving insight into and ovcisight of individual 
prognuus as well as l.)(.)l.)’s fohil portfolio of nuijor defenses ac.Tjuisitions. 'i’he 
collective perfonuance of the programs in DOD’s portfolio is a key indicator 
of how well the ac<}ui.sition .system generat i'S the return on investment that it 
promises to the warlightcr, Congress and taxpayers. GAO recently reportcjcl 
that, outcome melric-.s for D(.)D’s 2008 major ded'ense acriiuisition poitfolio 
show worsening pc-iibrnuuvoc when compared to the (lei>aitnient’s 2003 
portfolio. For example, total acquisition costs for programs in the 2008 
portfolio increaseti 25 percent from lli-sl estimates compared to a l9-porcent. 
increase for jirograms in the 2003 porifolio. DOD is working with (jAO and the 
Office of Managenu-rit and Budget to develop a coiuiu ehensive set of outcome 
metrics to better assess its portfolio of programs, 

While knowledge and outcome metrics provide valuable information about 
the potential problems and health of pri^grams, they are of limited valu(,i if 
DOD doc.s not do a belter job ensuring acquisitions begin with realistic plans 
and baselines prior to development start . (bAO believes there is a clear set of 
proreiiuisifes that must be met. by etic-h program's actiiiisitioti strategy before a 
measurement of the program’s health will be of real value, 'riicse prerixiuisites 
include: 1) establishing an evolutionary, knowledge-based business case for 
ca< h acquisition; 2) separating technology development from product 
dcveloinncnt; 3) Umiting time and requirements for product dcvclopmejit to 
manageable levels; 4) employing systc-ms (-ngineering early on in the pioct-ss 
to arrive at realistic cost and .schedule estimates; 5) coruniilting to fully 
fiiiuling a program once it is ajiprovcd; and 6) setting prioritic;s from the top tt.) 
ensure that candidate programs iire (niiy needed and have? a .solid plan for 
delivery. 
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Mr. Chairman anti Membei^ of the Committee; 

I am pleased to be tiere tills morning to disc uss how best to measure and 
detenninc whether DOD’s acquisition .system is providing value to the 
warUghter. Barlier this week, we reported that the cunuilative cost growtli 
in DOD’s portfolio of 96 major defense acfinisil ion programs was $296 
billion and the average delay in delivering promised capabilities to the 
warfighter was 22 months. Hiese outcomes meati that other critical 
defense and national priorities go unfiindcHi and wm-fighters go without 
the equipment they need to counter (‘ver changing threats that th(‘y face. 
This condition is unacceptable. VV'e believe that significant improvement in 
the acquisition of weapon sysKmis is possible and that the ability to 
measure knowledge, processes, and outcomes is crjtic;al l.o achieving that 
improvement.. It is important to note that not one sir\gle metric or scd. of 
metrics is enough to monitor acquisitions and gain el'ficiencies. Today, we 
would like to break our discussion about how to measure the department’s 
a(;(juisilions into 3 biistc sections; 

Pii-st, we would like to present a .set of motiics that we refer to as 
“knowledge metrics*’ and use to detenniiie how well acquisition programs 
arc managing and retiring predictable technology, design, and 
manufactming risks and gaining knowledge. These motxics are valuable 
because they can predic t problems and identify causes. 

Second, we would like to discuss a set of outcome measures — conceniing 
cost, schedule, and capability--that serve as health indicators. These 
indicators measure how well programs are being executed and achieving 
predicted out<-omes in lenns of meeting original baselines for cost, 
schedule, and perfoi iiiattce. Tho.se metrics have intrinsic value as simple 
measurements, just as a thermometer can warn a parent that a child lias a 
fevcu-. 


nhrd. there are certain indicatoi-s that we took for — based on the work we 
have d(.)ne examining best {)raeticos for pixKluct development — that arcN 
pprha[)s, more important that\ these Icnowledge and health metrics 
because they detenuine from the outset how realistic ihe acc^uisition plans 
a?Kl st rategies of programs are. For I hc .sake of t oday’s discussion, we will 
refer to them as “prerequisite indicators.” These prerequisites are ino.st 
important because we question tlu' value of ANY metric when rueasuiing 
from an imrealistic baseline. 

We know that the knowledge and program health metrics we use to 
measure’ programs’ progress and outcomes are valuable when used in 
realistic, miu'ket.-driven product development environments. We also kt\ow 


Viige I 


GAO-O‘t-343T 



54 


t fiat ALL of these metrics are imporLinl indicators f or dcKiision makers. 

Oiir extensive body of work examining worid-crlass enterprises and the 
way tiiey operate has validated their value for progratns that, must deliver a 
new pmduct to market at a certain time and within a certain investment 
c-ost or suffer significant consequences. Tiiese metrics work because they 
are measuring realistic plans an<l goals that are supported by doable 
requirements, accur^e cost tmd schedule estimates, and stable funding, 
'[’he com])any developing the products suffers dire consequences, sue!) as 
loss of market share, if these pntgrarus do not succeed. 

Tills statement draws from our extensive body of work on DOD’s 
acquisition of weapon systems. A list of our key products is provided at 
the end of this statement. 'I’his work wjis conducted in accordance with 
generally accepted government auditing standai'ds. 'I'hose standards 
require that we plan and perform the audit to obtain sufficient, appropriate 
evidence to provide a re<isonable basis for our findings and conclusions 
based on our audit objectives. We believe that the evidetice obtained 
provides a reasonable btisis for our findings and conclusions liased on our 
audit objectives. 


“Knowledge” Metrics 
Identify Potential 
Problems and Their 
Likely Causes 


We have conducted a body of work that examines weapon acquisition 
issues from a peispective that draws upon lessons learned from iiest 
practices in product development. Collectively, these practices comprise a 
process that is anchored in knowledge. Achieving the right knowledge at 
ttie right time enables leadei-ship to make informed decisions about when 
and how best to move into various expensive acquisition phases. In 
essence, knowledge siqiplants risk over time. This building of knowledge 
consists of gathering information about technology, design, and 
manufacturing at throe critIctU points over the course of a weapon system 
program (Figure 1). We have developed valuable “knowledge metxics” that 
measure this knowledge build and allow us to identify potential iiroblems 
that could lead to cost, schedule, or performance shorifixlls and their likely 
causes. I’he metrics c:an be doscribefi as 


• Knowlcdcie Point i, evidenced by the balance between a product’s 
required capabilities and the res<.)urces available to meet them. Focus 
should be on understanding technological and design implications and 
achieving a high level of technology maturity at the start of syst em 
development. This means that the critical technologies needed to meet 
essential product requirements must be demonst rated t o work in their 
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intended emironnient The technology reiuiiness level for each ci-ilical 
teclmology is the metric we use to nieasuix? technology matiirityJ 

• Knmvledge point 2, evidenced by the development of engineering 
prototypes and the completion of engineering drawings for an integrated 
product at the system design reriew. This metric prorides Umgible 
evidence that the product’s design is stable, meaning it luis a high 
probability of meeting customer requiremerds, as well as cost, schedule, 
and reliability Uu-gets. A best practice is to acrhieve design stability at the 
system-level critical design review, usually held midway through 
development. Completion of at least 90 percent of engineering drawings is 
the metric we use to measure design stability. 

• Knowledge point S, evulenced by the demonsti at ion that critical 
manufacturing processes are in control — that is, they are re}>eatable, 
sustainable, and capable of consLstently prodticing parts within the 
product’s quality tolerances and standards — at the start of production. One 
iumdred percent of critical manufact uring processes in <'ontrol is the 
ntetric we use to evaluate manufacturing maturity. 


Figure 1: Knowledge-Based Acquisition Process 
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Each of these metrics gauges the poim. when the requisite level of 
knowledge has been attained for a product in relation to where that 


' T*;rhnok>gy readiness levels, origiiwUy fle\«“lcipeti liy tiu- Nat ioiial A(‘i'cniaMt.i<‘s aiui Spare 
.'\dmini.sU‘ation, an' moasuiw! on a scale of 1 io9, bogitiMing u i!tt pajK'f siiiclies of a 
tcc)>nolof*y’s feasibility and culniinating with a ro<-hnoio!?y fully integraU'd im.o a c-omplel(?d 
prodiK-t. See Attac'hment 2 for the clcfinition.s of trchaology readiness levx'ts. 
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product is in its development. Wt>rld-<‘lass firms we have visited work liard 
to establish metrics such as these and their decision makers are required 
to consider them before deciding to advance a program to the next level. 
Tiieses types of metrics also help decision makei-s gauge progress in 
meeting cost, schedule, and performance goals and ensure' that managei s 
will (1) conducl activities to capture relevant product development 
knowledge, (2) provide evidence that, this knowledge has been crapturt'd, 
and (3) hold decision reviews to determine that appropriate knowledge 
has beer\ captured before moving the product to the next phase. The rc'sult 
is a product development process that provides critical measurements of 
knowle<Ige, holds decision makers accountable, and delivers the expe<ied 
results in a predictable manner. Attachment 1 to this statement provides a 
detailed list of activities that vvould provide program managers with the 
requisite technology, design, and manufacturing knowledge at key points 
ii\ tinte during development. 

We have used these metrics to identify problems on major w eapon system 
acquisitio!\ programs an<l have found a strong correlation between each of 
them and cost and schedule outcomes. For example, for 47 weapon 
programs in DOD’s 2008 port folio of major defense ac^ciiiisitions, we 
assevSvSed the knowledge attained at key decision points in the acquisition 
process and found the following:- 

Most programs have started system development without mature 
technologies. Only 4 of the 30 programs that: provided data on technical 
maturity at development start, did so with fully mature critical 
technologies. Rirtlier, only 14 of 39 programs that provided data have or 
plan to have demoirstrated all of their technologies in a realistic 
environment prior to system-level critical design review, at which point 
the system's design should be stable. The 5 newer programs — those 
irutiaied since 200o‘ — have higher !ev('ls of technology maturity, with all 5 


We diti (his tjy collfH-(}ng chila (Urecliy from |.<rognwii olTicos using i\ quostioniuiirc. Thc^sc: 
programs are primarily in tlcvolopniunt and, thviuioro- most rolovant If) l urronl dc'cisions 
about, \v-hich programs should WKX’ive substantial iin'o.sltnonts of resuart,:!) aiu! tlovoloinncnt. 
funding nowand lai^o amounts of procuiemeiU limUingin tiu' future. Defense 
AcquisUions: A'iscssvient of' lielecicd Weaimc-i DiXiQniriiN. (.iA* >.6!i.:t2(.iSf*. Wa.shingt-on, 

D.C.: March 30, 2009. 

'in 2003, DOD revised its primary acquisition policy to staio ihat technologies sltotilci bo 
rlcmonstrated in a relevant envi!x.>nm{'nt prior to starting an ac<)uis!tion [trogram. In 2006, 
ihi-s siandaril became a statutory rc<juircmcn( for all major defenst’ acxiuisilitjn ptxjgrams in 
the National Defense Authorization Act for Fiscal Year 2006, Pub. L .No. 100-163, ^ SOI. 
codiricd at 10 r.S.C. § 2366b. 
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pR>gisuns demonstrating tlieir technologies in a relevant environment 
prior to development start, in accordance with DOD and statutory criteria. 
However only lof these programs met. the best practice standard of 
demonstrating critical technologies in an operational envdronrnent. Last 
year, we detennined that programs with immature technologies at: the 
start of system development experieiKred 44 percent higher cost growth 
tt^an programs that began with matiir<‘ technologies. 

Programs that have held design reviews in recent years reported higher 
levels of design knowledge. However, designs, on average, are still far 
fnjm stable. For the 24 programs in our assessment that held a criticit! 
design review since 2003, the average* percentage of total expect ed design 
drawings releasable at this review was 65 percent, compared to a best 
practic-e standard of 90 percent. We liave found that programs moving 
forwanl into system demonstration with low leveivS of desigri stability are 
more likely than other programs to encounter costly design changes and 
parts shortages that, in turn, cause labor inefficiencies, schedule delays, 
and qualify problems. 

Altachsnenl 3 reprt'senfs our notional depiction of the problems and 
outcomes that cjm typically be expected when these knowledge metrics 
are followed vei-sus when they are not. Ger\eraliy speaking, progra,ms that 
move forward without retiring technology, design, and manufacturing risk 
at appropriate. iunctures will encounter a cascade of problems beginning 
with design changes and continuing with parts shortages, changes to 
manufacturing processes, lalror inefficiencies, and quality problems. All of 
these problems delay programs and add to their development costs. We 
have found, for example, that a significant portion — about 70 percent-of 
the total (levelopmcnt cost growth in programs t,ypicaliy occurs after the 
design review. 


Outcome Metrics 
Provide Insight into 
the Health and 
Performance of 
Individual Weapon 
System Programs and 
lioD’s Total Portfolio 


Program outcome metrics — t}uantitative measures of c'ost, schedule, and 
pci formancM', and changes in these factors over time — provide useful 
iiulu atoi's of the health of actiuLsition programs and facilitate analyses of 
how well programs are meeting cost, schedule, and peiforiuance goals. 
When assessed regularly fi)r changes and the reasons that caus(i! clianges, 
such iiulicatons can be valuable tcK>is for improving insight and overeight 
of individual programs as well as DOD’s total portfolio of major defense 
acquisitions. Over the yeai^s we have l eported co.sl , schedule and 
performance data — good and bad — on numerous weapon .systems. Our 
work continues to identify systemic and program-specific causes for cost, 
schedule, and i>ortormance problems and has led us to designate, since 
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1990, DOD’s management of mjyor weapon system acciuisitions as a high 
risk area. 


To improve acquisition perfonriance and address the i'a(-tors that keep 
weapon acquisitions on the high risk list, Dt)D is workii\g witii us and the 
Office of Management and Budget to develop a comprehensive set of 
outcome metrics to provide better, compreiK'nsive, and consistent 
measures of program cost and schedule performance. Last year, this 
cooperative effort resulted in agreement to track trends and changes in 
programs from their original baselines, from 5 years ago, and I'rom the 
previous yem*, for the following data points: 

• Development cost; 

• Ihocuremcnt cost; 

• i’otal },>rogram cost; 

• Quant ities to be procured; 

• Pi'ocurement unit costs; 

• Total program unit costs; 

• Cycle time from Milestone B to initial Operational Capability 

DOD initiated a pilot study of 7 mtyor defense programs (o assess the 
adequacy of the proposed metrics and their value in aruilyring 
porfoi'mance, and t.h(^ results proved promising. DOD appro\ ed tlie 
outcome metrics and intends to collect and report siic:l» data on an annual 
basis. Efforts to develof) similar metrics on schedule performance 
continue. 

We believe that the metrics DOT) plans to use are valuable for providing 
insight into tlie performance of weapon system programs. We have used 
similar metrics for many years in assessing programs. For example, we 
recently reported that ten of DOD’s hugest acquisition programs, 
commanding about half the overall acquisition dollars in the department’s 
2008 port.ff)lio of major programs, have ex'ijericnced significar\t cost 
growth and have seen quantities reduced by almost a third (see table 1). 
The two largest programs — the Joint Strike Fighter' and the Future Combat 
System — represent .significant. c:ost risk nroving forward and will dominate 
the portfolio for years. Since these pjograms consume such a large portion 
of the funding that DOD spends on research and development and 
procurement, their performatice also affects other nuyor weapon 
acquisitioas, smalicr acquisition programs, and DOD’s atiilit y to fund and 
acquire other supplies and equipment as well. 
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Table 1 ; Changes in Costs and Quantities for Ten of the Highest Cost Acquisition Programs 

Total cost 


(fiscal year 2009 dollars Acquisition 

in millions) Total quantity unit cost 


Program 

First full 
estimate 

Current 

estimate 

First full 
estimate 

Current 

estimate 

Percentage 

change 

Joint Strike Fighter 

206,410 

244,772 

2.866 

2.456 

38 

Future Combat System 

89,776 

129,731 

15 

15 

45 

Virginia Class Submarine 

58,378 

81,556 

30 

30 

40 

F-22A Raptor 

88,134 

73,723 

648 

184 

195 

C-17 Giobemaster III 

51,733 

73,571 

210 

190 

57 

V-22 Joint Services Advanced Vertical Lift Aircraft 

38,726 

55.544 

913 

458 

186 

F/A-iaE/F Super Hornet 

78,925 

51,787 

1.000 

493 

33 

Trident il Missile 

49,939 

49,614 

845 

561 

50 

CVN 21 Nuclear Aircraft Class Carrier 

34,360 

29,914 

3 

3 

-13 

P-8A Poseidon Multi-mission Maritime Aircraft 

29.974 

29.622 

115 

113 

1 


Source; GAO 000 iJaia. 


While program outcome metrics are good measures of individual program 
performance, the colitK live perfonnance ol' l)(')L)’s portfolio of nutjor 
defense acquisition i>rograms is a key iiulicator of how well the 
depailnient’s acquisition system generates the return on investmetit it 
pnjiuises to the waiTighltu-, Congress, and tlie taxpayer. Poitfolio ntetiics 
also provide senior leadei-s and Congress with a snapshot of tJie 
cumulative impact of curreitt investment decisions and poor program 
perfonnance on future budgets. In our annual assessment of selected 
weajton programs, we analyzed the performance of Dt!)!) programs at th(* 
portfolio level t)y comparing progratvv.s’ initial cost, schedule, and quant ity 
estimates to their current estimates, tiasod on data obtained from the 
Selected Acquisition Reports. This year’s cuntulalive results, reported 
earlier this week, ‘ are shown in table 2. 


’ GAO, Defense A('(]Uis)tinn.H: AsNCSs^niM'ila nj' SoUrtcil Wraprui Projii aiHK. ( i A( ) iiH 
{Washington, D.C.; Mar. 30, 2000). 
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Table 2; Analysis of DOD Ma|or Defense Acquisition Program Portfolios^ 


Fiscal year 2009 ddlars 

Fiscal Year 


2003 

2007 

2008 

Portfolio size 

Number of programs 

77 

95 

96 

Total planned commitments 

$1.2 

S1.6 

SI .6 

trillion 

trillion 

trillion 

Commitments outstanding 

$724.2 

$875.2 

S786,3 


billion 

billion 

billion 

Portfolio indicators 

Change to total RDT&E costs 

37 

40 

42 

from first estimate 

percent 

percent 

percent 

Change to total acquisition cost 

19 

26 

25 

from first estimate 

percent 

percent 

percent 

Total acquisition cost growth 

S183 

$301,3 

$296,4 

billion 

billion" 

billion 

Share of programs with 25 

41 

44 

42 

percent increase irt program 
acquisition unit cost growth 

percent 

percent 

percent 

Average schedule delay in 

18 

21 

22 

delivering initial capabilities 

months 

months 

months 


SoLiice,': SAO itnalysis o( DOO 4eta. 

“Data were obtained from DOD's Selected Acquisition Reports (dated December 2002, 2006, and 
2007). In a few cases dala were oblatned directly trom program offices. The number of programs 
reflects the programs with Selected Acquisition Reports; however, in our analysis we have broken a 
tew Selected Acquisition Reports programs into smaller elements or programs. Not all programs had 
comparable cost and schedule data and these programs were excluded from the analysis where 
appropriate. Portfolio performance data do not include cosls of developing Missile Defense Agency 
elements or the OIMHHS program, 

•The acquisition cost growth for the 2007 portfolio was $295 billion in 2008 constant dollars. 

Analyzing the data and conipanng inotncs ft oni different lime periods 
provides unique insights into the relative Itealth of the portfolio and 
trends. From 2003 to 2008, the number of programs in DOD’s mtyor 
defense actjiiisition poitfolio has grown from 77 to 96. T(.ttal crosls for tht^se 
programs now total $1.6 trillion with almost one-half of this amouttl still to 
be spent. Outcome metrics for 2008 show worsening perfonnance in all 
categoties compared to the 200.3 portfolio and mixed perforntance — some 
better, some worse — compared to the 2007 data. Wliile DOD is committing 
.substantiiiHy more investment dollajES to developing and procuring new 
weapon systems, the total acquisition costs for the 2008 port folio has 
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grown by $290 billion over initial estiiiiales and the average scliedule delay 
in delivering capabilities to tlie warfighter averages 22 months. 
Implications for the future are obvious. Cloi\linued cost growth reduces 
DOD’s buying power iind results in less funding being available for other 
DOi') priorities and programs. As program c osts inc'it'juse, DO!) must 
reciuest more funding to cover overruns, make? trade-offs witl\ existing 
pr(,>granis, delay the start of new programs, talce funds from other 
accounts, or reduce procurement quantities. Continued failure to deliver 
weapon systems on time delays providing critical capabilities to the 
wiuilglitcr iuid results in openitir\g costly legacy systems longer timn 
expected, finding alternatives to fill capability gaps, or going completely 
without the capability. 


While the metrics discussed above can provide? valuable knowledge about 
potential problems and additional information on the hoiilt fi of DOD's 
acquisition programs, met rics alone may not be sufficietYt if the 
dopariment does not do a better job ensuring that acrqiiisitions begin with 
realistic plans and baseline estimates for cost and schedules prior to 
development start. We believe there is a dear set of prerequisites that 
must be a |)art of any acqui.sit ion strategy before any mc^asurf'ment of tlie 
acquisition’s health can be valuable. Otherwise, metrics measured agtiinst 
unrealistic estimates will do no good. These key prereciuisites for 
obtaining realistic bjiselines include: 


Key Prerequisites 
Need to Be Met to 
Improve the Value of 
Metrics and Achieve 
Better Acquisition 
Program Outcomes 


• Esfabliahing a clear, kuowU’dge-based, eixdulkmarfj business case for the 
pivducl. 'I'his business case must; validate that a need exists; determine 
that resources are available to develop a product that will meet the ne(Ki; 
determine that the [u oduct developer has a knowledge-based phm and 
strategy to deliver- tht? pr oduct; establish rt'^sonable estimates for’ cost:, 
delivery time and quantities; and ensur-e available funding for the product. 
Ail of these elements of the business case should also be agreed upon by 
major stakeholders across the re<iuir'ements, funding, acquisition, and 
warfrghting lonirnunities. 

• Scparaling technology development acilvilicsfrom product devcloptneril 
acihntk’s. The process of developing tochnoiogy culminates in 
discovery — the gathering of knowledge — and must, by its nature, allow 
|•(X)m for unexpected results and delays. Leading finns do not ask their 
prcKluct rmmagers to develop te(.:hr-iolog.y because they have- leanred the 
hard way that invention carurot be scheduled. When immature 
technologies are brought onto the critical path of product development 
programs too early, they often cause long delays in an environment w here 
large workforces must be employed, complex tools, plants, mui facilities 


Page 9 


GAO-»i)-r>4:3'l' 



62 


must be operatetl, long and expensive sup[)lier ru't works must be paid, aiui 
the product itself must sometimes be redesigned once t he final form of the 
tetrhnologies is known. Successhil programs give responsibility for 
maturing technologies to science and technology organizations, rather 
than the program or product development managers, because the science 
and technology emdronment is less costly. We have recommended in the 
past that DOD’s risks should be taken in the science and technology arena 
and that more funding should be made available to tiiis procc'ss to do so. 

LimUing time and reqiiiremenl-s for product development to manageable 
levels. Product developers should strive to deliver the best available 
capabilities within realistic timeframc^s and should expect to continue lo 
dcvcdop new cap^ilities when they are technologically fc'asible. By 
limitittg product development cycle times to 6 years or less, DOD could 
assimilate new' tecrhnologies into weapon systems more freciuently, 
accelerate delivery of new teclmoiogy to the warfighter, hold program 
mattagej's accoiintal:)le, and make more frequent and predictable work in 
production, where contractors and the industrial base can profit by being 
efficient. Too tnany rn^jor acctuisitions currently take the opposite 
appro;,\<“li by sc'eking to deliver a revolutionmy “big bang” capability in one 
•Step. Tills means that programs are more risky, delivery takes tts long as 15 
years iti some cases, mid costs grow at exponential rates fi'orn the original 
baseline due to the risky nature of the acquisition strategy, We point to the 
private sector and some piist defense acquisitions, such as the P-16 
program, as models for this practice. 

Emgyloying earhi syalenis engineering discipline in order to develop 
realistic cost and .schedule estimates pnor to developnuml start. Early 
system.s engineering provides the knowledge a product developer needs to 
identify ami ri'soh e performance arul resource gaps before product 
(levelopinont begins either by reducing requirements, deferring them to 
tlic futu!-e, or increasing the estimated cost for the weapon system's 
development. Requirements that are too risky given the state of 
technology and design should not be allowed into this expensive 
environment. 

Making a commitment loJuUyjmul programs once they are approved. 
'I’his would require the department to ensure that, it does not liave too 
many progrmns underway given the amount of avtiiiable resources. 

Selling priontiesfrom Ihe top lo ensure that candidate programs arc 
Indy needed and have « solid- plan for ddiveiy. DOD will continue to 
experience poor acquisition outcomes until it begins making choices that 
reflect Ihe most important needs of the joint warfighter and inatdi 
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requiremente \vith a\'ailable resources. The urge to accept all candidate 
programs and to go for the “big btuig” capability without the knowledge to 
achieve it should be resisted. Only the best candidates-defined in terms of 
priorities, resource availability, and executability-should be approved. 

There is no doubt that the current state of the depaitment’s acxjuisition 
proce^ is too expensive for the taxpayer and not t imely erK)ugb for the 
waihghter. The following illustration reinforces tliis point. 


Figure 2: Cost Remaining Versus Annual Appropriations for Ma|or Defense Acquisitions 

Dollars (iri b'liio.-isj 
1,000 



1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 

Fiscal year 

} ] Cosis Hemaiojrig tor Major Detoosc Acquisitions 

ilBI Annual ROTE and Procuromont Appropriations 
Soutcn: OOD (liaia): GAO (analysis ana presantoiionv 

This figure depicts mt investment, strategy for major weapon systems that 
continues to increase the costs to develop our existing weapons well into 
the future while the funding available to retire those costs appears capped 
at a very low level. Wl\ile costs continue to rise as the result of more and 
more risky programs being added to the [)ortfoUo, our ability to allocate 
funds for these costs appears to l;>c, at. best, capptx! at veiy low 
percentages of the total cost. We could measure the risk of these 
acquisitions much better than wo have in the pfist if we set t he appropriate 
prerequisites for their initiation, measure tlie knowledge that must l^e in 
place at various points, and continue to monitor their health in tenus of 
cost and schedule. 


rage 1 1 


G.\0-()!)-543T 




64 


Concluding Remarks 


Measuring the perfom^ance of weapon system prijgninis both iiKlividiially 
and collectively is critical for deifinnining whether the waiTigliter ar\d the 
taxpayer iire receiving the promised return on investment. No single 
metric, however, can capture the whole piciure of how well programs are 
perfonning. It is importmit to look at knowledge and outcome metrics. 
ICnowletlge metrics provide key information for detennining wiiether 
programs have the requisite knowledge to mov(^ fiom one phase of 
development to tl\e next and are at risk of cost and schedule overruns. 

( Xitcomes metrics ai'e also needed to provide teinperiiture checks on the 
health and status of individmiJ programs and the portfolio of programs as a 
whole. These metrics are vital for infonning program decision making arid 
helping to manage programs. 


Metrics t)y themselves do i\ot solve problematic acquisitions. Ultimately, 
DOD still needs to do a better job planning and executing progranis to 
achieve better outcomes. (Mtical to achieving sitcc.-essful outcomes is 
establishing knowledge-based, realistic program baselines. Witliout 
realistic baselines, there is no foundation for accurately measuring tlie 
knowledge and health of programs. (.)ver the past several years, our work 
has highlighted a number r)f uruierlying causes for why DOD tkx^s not 
effectively maitage the acquisition of weaporr systoxn programs. DOT.) 
recently i-evis(‘d its acejuisition policy to provide a better fourKlation for 
developing weapor* systems, however, reform will not be achieved witho\it 
fuTtdamental chmiges to the overall a<'C|uisition culture and erwironment 
that t^xists in DOD. i would be pleased to discuss tlu'se causes aiid issues 
with tlio Committee at a future time. 


Mr. Chainnarg this c'oncKides my prepared statement. I would be happy to 
answer any questions yovi may have at this time. 


Contacts and 
Acknowl edgements 
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this statement include Cheryl Andrew, Ridge Bowman, Bruce Fairbairn, 
Susan Neill, John Oppenheim, and Ron Schwenn. 
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Attachment 1: Knowledge-Based Activities 


knowledge Point 1 : Start of product development activities 

Best practice metric; Technology readiness level 7 (indicating technologies work In an operational environment) 

Demonstrate technologies to high readiness levels 

Ensure that requirements for the product tncremeni are informed by preiimirrary design using systems engineering process (such as 
prototyping of preliminary design) 

Establish cost and schedule estimates for product on the basis of knowledge from preliminary design using system engineering tools 
(such as prototyping of preliminary design) 

Constrain development phase (5 to 6 years or less) for incremental development 
Ensure development phase fully funded (programmed in anticipation of milestone) 

Align program manager tenure to complete development phase 

Contract strategy that separates system integration and system demonstration activities 

Conduct independent cost estimate 

Conduct independent program assessment 

Conduct major milestone decision review for development start 

Knowledge Point 2: Transition from system integration to system demonstration activities 

Best practice metric; 30 percent of design drawings are complete by the critical design review 

Complete system criticai design review 

Complete 90 percent of engineering design drawing packages 

Complete subsystem and' system design reviews 

Demonstrate with system integration prototype that design meets requirements 
Complete the failure modes and effects analysis 
Identify key system characteristics 
Identify critical manufacturing processes 

Establish reliability targets and growth plan on the basis of demonstrated reliability rates of components and subsystems 
Conduct independent cost estimate 
Conduct independent program assessment 

Conduct major milestone decision review to enter system demonstration 
Knowledge Point 3: Initiation of producing a product to be delivered to customer 
Best practice metric: 100 percent of critical manufacturing processes are in control 
Demonstrate manufacturing processes 

Build and test production-representative prototypes )o demonstrate produci in operational environment 
T est production-representative prototypes to achieve reliability goal 
Collect statistical process control data 

Demonstrate that critical processes are capable and m statistical control 
Independent cost estimate 
Independent program assessment 

Conduct major milestone decision review to ^gin production 

Soijco GAO analysts oicotrvnanjiiBtiesi .ofaaKes. 
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Attachment 2 : Technology Readiness Levels 


■■ 





Technology readiness 
level 

Description 

Hardware 

Software 

Demonstration 

Environment 

1. 

Basic principles 
observed and 
reported. 

Lowest level of technology readiness. 
Scientific research begins to be translated 
into applied research and development. 
Examples might include paper studies of a 
technology's basic properties 

None {paper studies and 
analysis) 

None 

2. 

Technology concept 
and/or application 
formulated, 

Invention begins. Once basic principles are 
observed, practical applications can be 
invented. The application is speculative 
and there is no proof or detailed analysis to 
support the assumption. Examples are still 
limited to paper studies. 

None (paper studies and 
analysis) 

None 

3. 

Analytical and 
experimental critical 
function and/or 
characteristic proof 
of concept. 

Active research and development is 
initiated. This includes analytical studies 
and laboratory studies to physically 
validate analytical predictions of separate 
elements of the technology. Examples 
include components that are not yet 
integrated or representative. 

Analytical studies and 
demonstration of nonscale 
individual components (pieces of 
subsystem). 

Lab 

4. 

Component and/or 
breadboard. 
Validation in 
laboratory 
environment. 

Basic technological components are 
integrated to establish that the pieces will 
worK together. This is relatively “low 
fidelity" compared to the eventual system. 
Examples include integration of “ad hoc" 
hardware in a laboratory. 

Low fidelity breadboard. 

Integration of nonscale 
components to show pieces will 
work together. Not fully functional 
or form or fit but representative of 
technically feasible approach 
suitable for flight articles. 

Lab 

5. 

Component and/or 
breadboard 
validation in relevant 
environment. 

Fidelity of breadboard technology 
increases significantly. The basic 
technological components are integrated 
with reasonably realistic supporting 
elements so that the technology can be 
tested in a simulated environment. 
Examples include “high fidelity" laboratory 
integration of components. 

High fidelity breadboard. 
Funclionally equivalent but not 
necessarily form and/or fit (size 
weight, materials, etc). Should be 
approaching appropriate scale, 
May include integration of several 
components with reasonably 
realistic support 
elements/subsyslems to 
demonstrate functionality, 

Lab demonstrating 
functionality but not form 
and fit. May include flight 
demonstrating breadboard 
in surrogate aircraft. 
Technology ready for 
detailed design studies. 

6, 

System/subsyslem 
model or prototype 
demonstration in a 
relevant 
environment. 

Representative model or prototype system, 
which is well beyond the breadboard tested 
for TRL 5, is tested in a relevant 
environment. Represents a major step up 
in a technology’s demonstrated readiness. 
Examples include testing a prototype in a 
high fidelity laboratory environment or in 
simulated realistic environment. 

Prototype. 

Should be very close to form, fit 
and function. Probably includes 
the integration of many new 
components and realistic 
supporting eiements/subsystems 
if needed to demonstrate full 
functionality of the subsystem. 

High-fidelity lab 
demonstration or 
limited/restricted flight 
demonstration for a 
relevant environment. 
Integration of technology 
is well defined. 
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Technology readiness 
level 

Description 

Hardware 

Software 

Demonstration 

Environment 

7. System prototype 
demonstration in a 
realistic 
environment. 

Prototype near or at planned qaerational 
system. Represents a maior step up from 
TRL 6, requiring the demonstration of an 
actual system prototype in a realistic 
environment, such as in an aircraft, vehicle 
or space. Examples include testing the 
prototype in a test bed aircraft. 

Prototype. Should be form, fit and 
ftinction integrated with other key 
supporting elements/subsystems 
to demonstrate full functionality of 
subsystem. 

Flight demonstration in 
representative realistic 
environment such as 
flying test bed or 
demonstrator aircraft. 

Technology is well 
substantiated with test 
data. 

8. Actual system 

completed and "flight 
qualified" through 
test and 
demonstration. 

Technology has been proven to work in its 
final form and under expected conditions. 

In almost ail cases, this TRL represents the 
end of true system development. Examples 
include developmental test and evaluation 
of the system in its intended weapon 
system to determine if it meets design 
specifications. 

Flight qualified hardware 

Developmental Test and 
Evaluation (DT&E) in the 
actual system application 

9. Actual system "flight 
proven" through 
successful mission 
operations. 

Actual application of the technology in its 
final form and under mission conditions, 
such as those encountered in operationaf 
test and evaluation. In almost all cases, 
this is the end of the last “bug fixing" 
aspects of true system development. 
Examples include using the system under 
operational mission conditions. 

Actual system in final form 

Operational Test and 
Evaluation (OT&E) in 
operational mission 
conditions 
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Attachment 3: Notional Illustration Showin; 
the Different Paths That a Product’s 
Development Can Take 
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WITNESS RESPONSES TO QUESTIONS ASKED DURING 

THE HEARING 

April 1, 2009 




RESPONSES TO QUESTIONS SUBMITTED BY MR. ANDREWS 

Mr. Ahern. The version of DoD Instruction 5000.2 just issued in December 2008 
eliminates much of what others refer to as “wiggle room” compared to the 2003 
version. I view the Department’s task now as one of the Components properly exe- 
cuting their acquisition programs in accordance with DoDI 5000.2 and the OSD staff 
ensuring we undertake disciplined, thorough program and milestone reviews before 
allowing programs to proceed to the next acquisition phase. We will monitor how 
the new guidance is being applied and, if necessary, will issue policy changes to 
sharpen and improve the process. If any of our changes require new legislation, the 
Department will submit a legislative provision for your consideration. [See page 28.] 

Mr. Ahern. What you suggest is a SAR-like submission during the Technology 
Development phase of the acquisition process. Technology Development is a contin- 
uous technology discovery and development process reflecting close collaboration be- 
tween the S&T community, the user, and the system developer. It includes signifi- 
cant competitive prototyping that will inform us on the realism of requirements and 
the maturity of technology. It will also significantly improve our cost estimates for 
the Engineering and Manufacturing Development and the Production and Deploy- 
ment phases. However, at this point in the acquisition process there is no clearly 
defined program to report on. As such, an annual SAR-like submission that purports 
to provide SAR-quality information for the technology development effort’s life cycle 
would have limited credibility or utility. There is an existing certification require- 
ment (10 U.S.C. 2366a) that does establish expectations for system cost during tech- 
nology development. Under that provision, if the projected cost of the system, at any 
time prior to Milestone B approval, exceeds the cost estimate for the system sub- 
mitted at the time of the certification by at least 25 percent, the Milestone Decision 
Authority, in consultation with the Joint Requirements Oversight Council, shall de- 
termine whether the level of resources required to develop and procure the system 
remains consistent with the priority level assigned by the Joint Requirements Over- 
sight Council. The Milestone Decision Authority may then withdraw the certifi- 
cation concerned or rescind Milestone A approval if the Milestone Decision Author- 
ity determines that such action is in the interest of national defense. This is con- 
sistent with the iterative nature of the Technology Development phase in assessing 
the viability of technologies while simultaneously refining user requirements allow- 
ing the Milestone Decision Authority to make an informed judgment whether the 
priority warrants committing to a higher-cost program. Once the program has been 
initiated and receives Milestone B approval, the Department establishes an Acquisi- 
tion Program Baseline, holds the program manager accountable for execution to it, 
and provides Congress the SAR. [See page 28.] 

Mr. Sullivan. A pre-Milestone B SAR-like report could be a valuable tool for as- 
sessing whether a program is on track to have a solid understanding of require- 
ments, technology, and cost before formally becoming an acquisition program. It 
could serve as the basis for illuminating early trades in all areas (cost, schedule, 
requirements, and technology) and as an early warning mechanism to identify pro- 
grams that are proceeding without the requisite knowledge in those areas. However, 
given that the pre-Milestone B technology development phase is a time when cost, 
schedule, and performance trades should be encouraged, we would not recommend 
using this reporting tool as a baseline control mechanism to apply Nunn-McCurdy- 
like standards to technology development costs. Annual reporting could begin at 
Milestone A. Much like the current SAR, it could provide basic information about 
the mission need the program fulfills the acquisition and technology development 
strategies, the program’s activities to date, and contract performance. It could also 
provide information on the programs’ schedule, cost, performance, and knowledge 
that is tailored to the early stages of the acquisition process. These data could in- 
clude the following: 

Capability Need: A description of the capability need that justifies the program, 
including the following: 

• capability gap that needs to be filled 

• priority level assigned by JROC to this capability need 

(77) 
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• timeframe in which the overall capability is required 

• type of materiel solution preferred — information system solution, evolutionary 
development of an existing capability, or a transformational approach 

Analysis of Alternatives (AOA): A description of the assessment and results, 
including the following: 

• the scope of alternatives considered in the AOA 

• the recommended solution derived from the AOA 

• the technical, operational, and programmatic risks identified with the rec- 
ommended solution 

Schedule: A baseline estimate set at Milestone A and current estimates for the 
completion of the following: 

• Systems engineering reviews: System Functional Review, System Requirements 
Review, Software Specification Review, Preliminary Design Review 

• Technology development: Technology Readiness Assessment, Prototype Dem- 
onstration (start and completion) 

• Requirements: Capability Development Document 

• Development cycle: Estimated cycle time in months (Milestone A to B), Esti- 
mated cycle time in months (Milestone B to C) by increment of capability (if 
applicable) 

Cost: A baseline estimate set at Milestone A and current estimates in base year 
and then year dollars for the following: 

• Cost estimate for Milestone B through completion reported as a range of likely 
costs 

Performance: Prioritized list of Key Performance Parameters that includes: 

• Proposed performance baseline at Milestone A 

• Current estimate of performance 

• Level of performance that will be demonstrated in the Technology Development 
Phase 

• Level of performance that has been demonstrated in the Technology Develop- 
ment Phase 

• Critical technologies that are enablers for each Key Performance Parameter 

• Description of requirements that were added or removed during the Technology 
Demonstration Phase 

Technology knowledge: List of the program’s critical technologies that includes: 

• Milestone A, current, and projected Milestone B technology readiness levels 

• Most current test environment (lab, relevant, operational) 

• Most current physical status (breadboard, functional prototype, full-up proto- 
type) 

• Description of trades available if technology does not mature as planned (use 
an existing technology, reduce or defer requirements, etc) 

• Schedule for maturing technologies to TRL level 7 (i.e. demonstrated in a real- 
istic environment) 

Design knowledge: Current estimates of the following: 

• Total and projected number of drawings released by the Preliminary Design Re- 
view 

• Estimated size of the software development effort (in lines of codes) 

[See page 28.] 


RESPONSE TO QUESTION SUBMITTED BY MR. CONAWAY 

Mr. Sullivan. DOD policy requires the CAIG to prepare an independent life cycle 
cost estimate for a major defense acquisition program Milestone B decision. The pol- 
icy states that the Milestone Decision Authority shall consider the CAIG estimate 
before approving the program to start system development. It does not require a rec- 
onciliation of the CAIG estimate with other service or program office estimates. 

In a July 2008 report, GAO found that program cost estimates are often signifi- 
cantly understated — a finding consistent with cost growth patterns reported by 
RAND, the Institute for Defense Analysis (IDA), and other organizations that con- 
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duct defense analyses. ^ In that report, GAO analyzed the cost of 20 major defense 
acquisition programs through December 2007. While the CAIG estimates generally 
underestimated costs by a smaller amount than program office and service esti- 
mates, the CAIG estimates could underestimate a program’s costs by billions of dol- 
lars (see table 1). For example, the initial service estimate for the development of 
the Marines’ Expeditionary Fighting Vehicle was about $1.1 billion. The CAiG esti- 
mated the development cost of the program to be $1.4 billion, but the expected de- 
velopment costs for the program had grown to close to $3.6 billion. In the case of 
the Future Combat System (FCS), the Army’s initial estimate for the development 
cost was about $20 billion, while the CAIG’s estimate was $28 billion. DOD began 
the program using the Army’s estimate of $20 billion, but development costs for FCS 
had grown to an estimated $28 billion. Many programs are also approved to start 
development based on the service or program office cost estimate rather than the 
CAIG estimate. Less than a quarter of the 48 programs in GAO’s 2009 assessment 
of weapon system programs that provided data used the estimate made by the CAIG 
as a basis for the program’s baseline, while almost 70 percent of the programs used 
the program office or service cost estimate.^ [See page 27.] 

Tabic 1 : Development Cost Estimates and Baselines for 20 Major Weapon System Programs 

(2008 dollars in millions) 


Program 

Development cost estimate 

Development cost baselines 


Service 

CAIG 

Percent 

difference 

Initial 

As of 
December 
2007 

Percent 

change 

Global Hawk 

S905 

$992 

10% 

$967 

$3,515 

264% 

UH-60M helicopter upgrade 

311 

379 

22 ; 

311 

838 

169 

WIN-T 

338'’ 

807 

139 

796 

2,088 

162 

C-130 Avionics Modemi 2 ation 

1,020 

1,175 

15 

720 

1,844 

156 

Expeditionary Fighting Vehicle 

1,056 

1,438 

36 

1,472 

3.556 

142 

Advanced Extremely High 
Frequency Satellites 

3,031 

3,175 

5 

2,923 

6,008 

105 

Wideband Global SATCOM 

296 

414 

40 

199 

323 

62 

Future Combat Systems 

20,248 

27,184 

34 

20,248 

27.955 

38 

Joint Strike Fighter 

30,500 

31,476 

3 

33,939 

40.210 

18 

COBRA JUDY replacement 

1,398 

1,521 

9 

1,527 

1,626 

6 

E-2 Advanced Hawkeye 

3,495 

3,720 

6 

3,589 

3,796 

6 

EA-18G 

1,707 

1,795 

5 

1,797 

1.865 

4 

VH-71 Presidential Helicopter 

3,378 

3,569 

6 

3,771 

3.859 

2 

Joint Land Attack Cruise Missile 
Defense Elevated Netted Sensor 

1,781 

1,926 

8 

1,894 

1,922 

1 

C-5 Reliability Enhancement and 
Engineering Program 

1,454 

1,583 

9 

1,627 

1,630 

0" 

CH-53K Heavy Lift Replacement 

3,970 

4,293 

8 

4,149 

4,095 

-1 

Longbow Apache III 

1,155 

1,382 

20 

1,095 

1,087 

-] 

Multi-mission Maritime Aircraft 

6,100 

6,970 

14 

7,080 

6,804 

-4 

Small Diameter Bomb 

416 

427 

3 

415 

395 

-5 

Standard Missile 6 

1,000 

992 

-1 

1,009 

932 

-8 


Source: GAO analysis of DOD data. 


''The semce's original estimate for tlie WIN-T program was $363 million but was revised prior to Milestone B to 
align with the CAIG's estimate. 

data include Air Force and Navy portions of the program only. 

'^C-5 RERP costs have increased by almost S4 million. However, due to rounding, the tabic indicates a 0 percent 
increase. 


^GAO, Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major 
Weapon System Program Outcomes, GAO-08— 619 (Washington. D.C.: July 2, 2008); Assessment 
Panel of the Defense Acquisition Performance Assessment Project for the Deputy Secretary of 
Defense, Defense Acquisition Performance Assessment Report (Washington, D.C.: Jan. 2006): De- 
fense Science Board, Defense Science Board Summer Study on Transformation: A Progress As- 
sessment (Washington, D.C.: Feb. 2006); RAND, Historical Cost Growth of Completed Weapon 
System Programs (Santa Monica: 2006); and Institute for Defense Analysis, Costs Growth in 
Major Weapon Procurement Programs, Presentation to the 38th Annual DoD Cost Analysis Sym- 
posium (Williamsburg: Feb. 2005). 

2 GAO. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO— 09— 326SP 
(Washington, D.C.: March 30, 2009). 
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RESPONSE TO QUESTION SUBMITTED BY MR. SESTAK 

Mr. Ahern. The Department addresses Modeling and Simulation (M&S) 
Verification, Validation, and Accreditation (W&A) both in policy and in guidance. 
On the policy side, DoD Instruction 5000.2, “Operation of the Defense Acquisition 
System,” addresses M&S as part of an integrated test and evaluation (T&E) con- 
tinuum that includes developmental, operational, and live fire T&E; family-of-sys- 
tems interoperability testing; and information assurance testing. The Test and Eval- 
uation Strategy requires empirical data to validate models and simulations and ex- 
pects reconciliation of pre-test predictions with post-test results. The policy also pro- 
vides for the use of accredited models in support of developmental T&E, initial oper- 
ational T&E, and live fire T&E. DoD Instruction 5000.59, “DoD Modeling and Sim- 
ulation (M&S) Management,” establishes the M&S Steering Committee that over- 
sees the development of VV&A policies, plans, and procedures. DoD Instruction 
5000.61, DoD “Modeling & Simulation Verification, Validation and Accreditation,” 
establishes common-sense guidelines and requires that models and simulations used 
to support major DoD decision-making organizations and processes (e.g., the De- 
fense Acquisition Board; Joint Requirements Oversight Council; and Planning, Pro- 
gramming, Budgeting, and Execution System) “shall be accredited for that specific 
purpose by the M&S application sponsor”. DoD Instruction 5000.61 also requires 
that W&A be documented. In terms of guidance, the Department has taken a num- 
ber of steps. The Defense Acquisition Program Support Methodology used in pro- 
gram support reviews includes strong criteria for evaluating a program’s W&A ef- 
forts. We have an on-line “W&A Recommended Practices Guide.” A new military 
standard, MII^STD 3022, “Documentation of W&A for M&S” was approved last 
year and is already in use for acquisition purposes across the Department. A DoD 
W&A Documentation Tool automates production of the MII^STD 3022 W&A doc- 
ument set and became operational this year. We are also developing risk-based 
W&A guidelines and pursuing routine examination of W&A when M&S informs 
major acquisition decisions. The Department does not keep central records of W&A 
and no studies have been performed to assess W&A documentation. So, without a 
data call to the DoD Components, it is not practical to provide a quantitative assess- 
ment of overall DoD W&A performance. The Department recognizes that W&A is 
important so that we have confidence in our models and simulations. While W&A 
is covered both in policy and guidance, we also know that we need to continue work- 
ing with the Components to ensure models used in our decision-making processes 
are properly accredited. [See page 24.] 
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